Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
2026-03-17T21:18:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-17T21:18:30Z Disabled control of system clock
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20260317 21:18:30.451022 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.126:40107
--webserver_interface=127.6.122.126
--webserver_port=0
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260317 21:18:30.779232 6642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:30.779918 6642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:30.780441 6642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:18:30.814324 6642 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:18:30.814654 6642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:30.815284 6642 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:18:30.815562 6642 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:18:30.854460 6642 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.126:40107
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.122.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:30.856127 6642 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:30.858034 6642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:18:30.871016 6648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:32.274832 6647 debug-util.cc:398] Leaking SignalData structure 0x7b080002eda0 after lost signal to thread 6642
W20260317 21:18:32.760967 6642 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.890s user 0.769s sys 1.120s
W20260317 21:18:32.761350 6642 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.891s user 0.769s sys 1.120s
W20260317 21:18:30.871156 6649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:32.764240 6651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:32.767737 6650 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1890 milliseconds
I20260317 21:18:32.767807 6642 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:18:32.769400 6642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:18:32.772339 6642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:32.773816 6642 hybrid_clock.cc:648] HybridClock initialized: now 1773782312773743 us; error 80 us; skew 500 ppm
I20260317 21:18:32.774760 6642 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:32.781525 6642 webserver.cc:492] Webserver started at http://127.6.122.126:42529/ using document root <none> and password file <none>
I20260317 21:18:32.782793 6642 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:32.783125 6642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:32.783639 6642 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:18:32.788940 6642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data/instance:
uuid: "31726fd7f40442e49b2fe0f880793206"
format_stamp: "Formatted at 2026-03-17 21:18:32 on dist-test-slave-979h"
I20260317 21:18:32.790293 6642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal/instance:
uuid: "31726fd7f40442e49b2fe0f880793206"
format_stamp: "Formatted at 2026-03-17 21:18:32 on dist-test-slave-979h"
I20260317 21:18:32.798900 6642 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260317 21:18:32.805464 6658 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:32.806624 6642 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260317 21:18:32.807041 6642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal
uuid: "31726fd7f40442e49b2fe0f880793206"
format_stamp: "Formatted at 2026-03-17 21:18:32 on dist-test-slave-979h"
I20260317 21:18:32.807415 6642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:18:32.862035 6642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:18:32.863653 6642 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:18:32.864212 6642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:18:32.944098 6642 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.126:40107
I20260317 21:18:32.944134 6709 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.126:40107 every 8 connection(s)
I20260317 21:18:32.947743 6642 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
I20260317 21:18:32.951187 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 6642
I20260317 21:18:32.951732 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-0/wal/instance
I20260317 21:18:32.955823 6710 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:32.957546 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.125:32779
--webserver_interface=127.6.122.125
--webserver_port=0
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260317 21:18:32.970798 6710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:18:33.000706 6710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
W20260317 21:18:33.003373 6661 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
W20260317 21:18:33.008188 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
I20260317 21:18:33.057878 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 1
W20260317 21:18:33.062805 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
I20260317 21:18:33.131387 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 2
W20260317 21:18:33.136173 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
I20260317 21:18:33.227746 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 3
W20260317 21:18:33.233286 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
W20260317 21:18:33.312155 6712 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:33.312846 6712 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:33.313274 6712 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:18:33.348379 6712 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:18:33.348990 6712 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:33.349326 6712 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:18:33.349536 6712 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:18:33.376814 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 4
W20260317 21:18:33.381657 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
I20260317 21:18:33.391450 6712 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.125:32779
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.122.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:33.393024 6712 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:33.395032 6712 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:18:33.407253 6721 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:33.681187 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 5
W20260317 21:18:33.688982 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
I20260317 21:18:34.236719 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 6
W20260317 21:18:34.249177 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:32779: Network error: Client connection negotiation failed: client connection to 127.6.122.125:32779: connect: Connection refused (error 111)
W20260317 21:18:33.407856 6722 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:34.692075 6724 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:34.696082 6723 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1282 milliseconds
I20260317 21:18:34.696172 6712 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:18:34.697538 6712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:18:34.700486 6712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:34.702040 6712 hybrid_clock.cc:648] HybridClock initialized: now 1773782314701958 us; error 124 us; skew 500 ppm
I20260317 21:18:34.703011 6712 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:34.710990 6712 webserver.cc:492] Webserver started at http://127.6.122.125:43143/ using document root <none> and password file <none>
I20260317 21:18:34.712095 6712 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:34.712466 6712 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:34.713176 6712 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:18:34.717986 6712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data/instance:
uuid: "21186091f2c9410ea6e585c5ef2c6291"
format_stamp: "Formatted at 2026-03-17 21:18:34 on dist-test-slave-979h"
I20260317 21:18:34.719295 6712 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal/instance:
uuid: "21186091f2c9410ea6e585c5ef2c6291"
format_stamp: "Formatted at 2026-03-17 21:18:34 on dist-test-slave-979h"
I20260317 21:18:34.728937 6712 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.011s sys 0.001s
I20260317 21:18:34.735792 6733 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:34.737419 6712 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260317 21:18:34.737933 6712 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal
uuid: "21186091f2c9410ea6e585c5ef2c6291"
format_stamp: "Formatted at 2026-03-17 21:18:34 on dist-test-slave-979h"
I20260317 21:18:34.738286 6712 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:18:34.800025 6712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:18:34.802206 6712 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:18:34.802945 6712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:18:34.886307 6712 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.125:32779
I20260317 21:18:34.886389 6784 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.125:32779 every 8 connection(s)
I20260317 21:18:34.889436 6712 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/data/info.pb
I20260317 21:18:34.894421 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 6712
I20260317 21:18:34.894856 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-1/wal/instance
I20260317 21:18:34.896212 6785 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:34.897280 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:38857
--webserver_interface=127.6.122.124
--webserver_port=0
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260317 21:18:34.918678 6785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:18:34.952466 6785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:18:34.973486 6785 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } has no permanent_uuid. Determining permanent_uuid...
W20260317 21:18:34.976105 6737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
W20260317 21:18:34.980790 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.030539 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 1
W20260317 21:18:35.035403 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.103973 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 2
W20260317 21:18:35.109048 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.200577 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 3
W20260317 21:18:35.205732 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
W20260317 21:18:35.277283 6787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:35.277916 6787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:35.278468 6787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260317 21:18:35.309898 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } attempt: 7
W20260317 21:18:35.315296 6787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:18:35.315693 6787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:35.316002 6787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:18:35.316279 6787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:18:35.323678 6710 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } has no permanent_uuid. Determining permanent_uuid...
W20260317 21:18:35.330166 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.349472 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 4
W20260317 21:18:35.354512 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.359839 6787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.122.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:35.361197 6787 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:35.362838 6787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:18:35.375334 6799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:35.388758 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 1
W20260317 21:18:35.393206 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.474763 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 2
W20260317 21:18:35.481266 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.566730 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 3
W20260317 21:18:35.571189 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.654066 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 5
W20260317 21:18:35.658378 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:35.711706 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 4
W20260317 21:18:35.720999 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:36.004609 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 5
W20260317 21:18:36.011567 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:36.206132 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 6
W20260317 21:18:36.213831 6785 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:36.564163 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 6
W20260317 21:18:36.571209 6710 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:38857: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
W20260317 21:18:35.376109 6800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:35.381211 6802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:36.710116 6801 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection time-out
I20260317 21:18:36.710139 6787 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:18:36.727923 6787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:18:36.730779 6787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:36.732226 6787 hybrid_clock.cc:648] HybridClock initialized: now 1773782316732204 us; error 81 us; skew 500 ppm
I20260317 21:18:36.733176 6787 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:36.740782 6787 webserver.cc:492] Webserver started at http://127.6.122.124:40359/ using document root <none> and password file <none>
I20260317 21:18:36.741860 6787 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:36.742162 6787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:36.742709 6787 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:18:36.747866 6787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/instance:
uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
format_stamp: "Formatted at 2026-03-17 21:18:36 on dist-test-slave-979h"
I20260317 21:18:36.749274 6787 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal/instance:
uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
format_stamp: "Formatted at 2026-03-17 21:18:36 on dist-test-slave-979h"
I20260317 21:18:36.757794 6787 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260317 21:18:36.766448 6813 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:36.767786 6787 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260317 21:18:36.768296 6787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
format_stamp: "Formatted at 2026-03-17 21:18:36 on dist-test-slave-979h"
I20260317 21:18:36.768715 6787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:18:36.848493 6787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:18:36.850154 6787 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:18:36.850762 6787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:18:36.935756 6864 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.124:38857 every 8 connection(s)
I20260317 21:18:36.935791 6787 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.124:38857
I20260317 21:18:36.939039 6787 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
I20260317 21:18:36.941579 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 6787
I20260317 21:18:36.942098 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal/instance
I20260317 21:18:36.947590 6865 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:36.969272 6865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:18:37.008809 6865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:18:37.021646 6865 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:18:37.044252 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Bootstrap starting.
I20260317 21:18:37.052006 6865 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:37.054005 6865 log.cc:826] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Log is configured to *not* fsync() on all Append() calls
I20260317 21:18:37.059937 6865 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: No bootstrap required, opened a new log
I20260317 21:18:37.081941 6865 raft_consensus.cc:359] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.082618 6865 raft_consensus.cc:385] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:37.082824 6865 raft_consensus.cc:740] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e1c983d0224428a9a7b147a0e5e4ef, State: Initialized, Role: FOLLOWER
I20260317 21:18:37.083524 6865 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.086129 6876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.087055 6876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:37.088076 6865 sys_catalog.cc:565] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: configured and running, proceeding with master startup.
W20260317 21:18:37.106980 6819 tablet.cc:2379] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260317 21:18:37.129441 6887 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:18:37.129877 6887 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:18:37.149118 6876 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:37.123713 6787 master_runner.cc:186] Error getting master registration for 127.6.122.126:40107: OK, instance_id { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" instance_seqno: 1773782312921949 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260317 21:18:37.149729 6876 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.151050 6787 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:18:37.154407 6876 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:18:37.274483 6785 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 7
I20260317 21:18:37.301158 6785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Bootstrap starting.
I20260317 21:18:37.313656 6785 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:37.316358 6785 log.cc:826] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Log is configured to *not* fsync() on all Append() calls
I20260317 21:18:37.323112 6785 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: No bootstrap required, opened a new log
I20260317 21:18:37.356596 6785 raft_consensus.cc:359] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.357606 6785 raft_consensus.cc:385] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:37.357928 6785 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Initialized, Role: FOLLOWER
I20260317 21:18:37.358974 6785 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.362861 6891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.363782 6891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:37.365554 6785 sys_catalog.cc:565] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:18:37.365713 6760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:37.366751 6760 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e1c983d0224428a9a7b147a0e5e4ef in term 0.
I20260317 21:18:37.374358 6816 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:18:37.375949 6876 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:37.376394 6876 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:37.376778 6876 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:37.386574 6876 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.393285 6760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:18:37.394608 6760 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:37.392313 6876 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 1 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:18:37.415869 6760 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 47e1c983d0224428a9a7b147a0e5e4ef in term 1.
I20260317 21:18:37.418543 6816 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:18:37.419886 6876 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:37.422791 6876 raft_consensus.cc:697] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 1 LEADER]: Becoming Leader. State: Replica: 47e1c983d0224428a9a7b147a0e5e4ef, State: Running, Role: LEADER
I20260317 21:18:37.435053 6876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.470983 6897 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e1c983d0224428a9a7b147a0e5e4ef. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.472278 6897 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: LEADER
I20260317 21:18:37.479697 6906 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260317 21:18:37.488328 6903 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:18:37.488777 6903 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:18:37.448925 6712 master_runner.cc:186] Error getting master registration for 127.6.122.126:40107: OK, instance_id { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" instance_seqno: 1773782312921949 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260317 21:18:37.522490 6712 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:18:37.528191 6906 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:18:37.562973 6760 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Refusing update from remote peer 47e1c983d0224428a9a7b147a0e5e4ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260317 21:18:37.564373 6739 tablet.cc:2379] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:18:37.568288 6897 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:37.604691 6710 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } attempt: 7
I20260317 21:18:37.609913 6907 mvcc.cc:204] Tried to move back new op lower bound from 7265412372720779264 to 7265412372261052416. Current Snapshot: MvccSnapshot[applied={T|T < 7265412372720779264}]
I20260317 21:18:37.621461 6891 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e1c983d0224428a9a7b147a0e5e4ef. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.623224 6891 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:37.633630 6876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.636713 6876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: LEADER
I20260317 21:18:37.645421 6906 catalog_manager.cc:1357] Generated new cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:18:37.645761 6906 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:18:37.653466 6710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Bootstrap starting.
I20260317 21:18:37.666162 6912 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.667062 6912 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:37.668581 6710 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:37.680675 6710 log.cc:826] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Log is configured to *not* fsync() on all Append() calls
I20260317 21:18:37.697914 6906 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 21:18:37.698305 6710 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: No bootstrap required, opened a new log
I20260317 21:18:37.700022 6906 catalog_manager.cc:1514] Loading token signing keys...
W20260317 21:18:37.704555 6816 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.6.122.126:40107 timed out after 0.546s (SENT)
I20260317 21:18:37.775290 6906 catalog_manager.cc:6044] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Generated new TSK 0
I20260317 21:18:37.779654 6906 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:18:37.780088 6710 raft_consensus.cc:359] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.786465 6710 raft_consensus.cc:385] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:37.787173 6710 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31726fd7f40442e49b2fe0f880793206, State: Initialized, Role: FOLLOWER
I20260317 21:18:37.789249 6710 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.809911 6916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.810894 6916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:37.813495 6710 sys_catalog.cc:565] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:18:37.813652 6682 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:37.814591 6685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
W20260317 21:18:37.821321 6685 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:57870 (request call id 2) took 666 ms (client timeout 546 ms). Trace:
W20260317 21:18:37.821861 6685 rpcz_store.cc:269] 0317 21:18:37.154929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:37.155133 (+ 204us) service_pool.cc:225] Handling call
0317 21:18:37.821223 (+666090us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:18:37.815295 6684 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:18:37.818043 6683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:37.833549 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.122.65:0
--local_ip_for_outbound_sockets=127.6.122.65
--webserver_interface=127.6.122.65
--webserver_port=0
--tserver_master_addrs=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260317 21:18:37.836463 6683 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Refusing update from remote peer 47e1c983d0224428a9a7b147a0e5e4ef: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:18:37.837445 6916 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:37.838007 6916 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:37.839184 6876 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [LEADER]: Connected to new peer: Peer: permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:37.853654 6916 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:37.855019 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:37.854844 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:37.857227 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
W20260317 21:18:37.877651 6664 tablet.cc:2379] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:18:37.926209 6918 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:37.931284 6916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47e1c983d0224428a9a7b147a0e5e4ef. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.931478 6925 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:37.932284 6925 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:37.932199 6916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:38.046139 6934 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260317 21:18:38.287448 6935 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:38.288343 6935 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:38.292900 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:38.296602 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:38.298312 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:38.304065 6935 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:38.305620 6935 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:38.343607 6876 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:38.345925 6876 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: LEADER
I20260317 21:18:38.496167 6903 catalog_manager.cc:1269] Loaded cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:18:38.496603 6903 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: loading cluster ID for follower catalog manager: success
I20260317 21:18:38.507746 6903 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: acquiring CA information for follower catalog manager: success
I20260317 21:18:38.516048 6903 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260317 21:18:38.540179 6919 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:38.541087 6919 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:38.542219 6919 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:18:38.610067 6919 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:38.611915 6919 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.122.65
I20260317 21:18:38.687068 6919 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.122.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.122.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.122.65
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:38.688880 6919 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:38.691422 6919 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:18:38.727223 6946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:38.866824 6952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:38.868085 6952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:38.876391 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:38.881887 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:38.886442 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:38.892438 6952 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:38.893702 6952 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:39.075615 6934 catalog_manager.cc:1269] Loaded cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:18:39.077577 6934 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: loading cluster ID for follower catalog manager: success
I20260317 21:18:39.120486 6934 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: acquiring CA information for follower catalog manager: success
I20260317 21:18:39.131470 6934 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260317 21:18:39.377100 6955 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:39.387624 6955 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:39.395730 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:39.400036 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:39.404361 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:39.410102 6955 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:39.411242 6955 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:39.761598 6957 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:39.769057 6957 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:39.772858 6957 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:39.775400 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:39.776858 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:39.780179 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:39.781579 6957 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:40.360950 6958 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:40.361660 6958 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:40.364522 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:40.368992 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:40.371675 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:40.377328 6958 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:40.378683 6958 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:40.833611 6962 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:40.843815 6962 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:40.849001 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:40.853047 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:40.855021 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:40.859999 6962 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:40.861102 6962 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:40.965902 6964 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:40.966650 6964 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:40.969877 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:40.978410 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:40.980170 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:40.985458 6964 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:40.986789 6964 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:41.492043 6965 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:41.492553 6965 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:41.494549 6965 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:41.495465 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:41.496716 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:41.498472 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:41.499681 6965 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:18:38.750232 6949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:38.737732 6947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:41.559907 6948 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 2838 milliseconds
I20260317 21:18:41.560076 6919 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:18:41.561839 6919 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:18:41.565586 6919 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:41.567369 6919 hybrid_clock.cc:648] HybridClock initialized: now 1773782321567308 us; error 70 us; skew 500 ppm
I20260317 21:18:41.568547 6919 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:41.581259 6919 webserver.cc:492] Webserver started at http://127.6.122.65:40445/ using document root <none> and password file <none>
I20260317 21:18:41.582633 6919 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:41.582962 6919 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:41.583580 6919 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:18:41.591549 6919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data/instance:
uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
format_stamp: "Formatted at 2026-03-17 21:18:41 on dist-test-slave-979h"
I20260317 21:18:41.593366 6919 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/instance:
uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
format_stamp: "Formatted at 2026-03-17 21:18:41 on dist-test-slave-979h"
I20260317 21:18:41.613781 6919 fs_manager.cc:696] Time spent creating directory manager: real 0.019s user 0.010s sys 0.008s
I20260317 21:18:41.624033 6970 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:41.626619 6919 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.002s sys 0.005s
I20260317 21:18:41.627362 6919 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal
uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
format_stamp: "Formatted at 2026-03-17 21:18:41 on dist-test-slave-979h"
I20260317 21:18:41.629185 6919 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:18:41.705204 6919 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:18:41.708284 6919 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:18:41.709076 6919 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:18:41.714448 6919 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:18:41.721715 6919 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:18:41.721961 6919 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:41.722316 6919 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:18:41.722477 6919 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:42.011437 6919 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.65:44199
I20260317 21:18:42.011538 7082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.65:44199 every 8 connection(s)
I20260317 21:18:42.015640 6919 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/data/info.pb
I20260317 21:18:42.023015 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 6919
I20260317 21:18:42.023763 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/instance
I20260317 21:18:42.053879 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.122.66:0
--local_ip_for_outbound_sockets=127.6.122.66
--webserver_interface=127.6.122.66
--webserver_port=0
--tserver_master_addrs=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260317 21:18:42.114372 7083 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:18:42.114990 7083 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:42.116429 7083 heartbeater.cc:507] Master 127.6.122.124:38857 requested a full tablet report, sending...
I20260317 21:18:42.120204 6830 ts_manager.cc:194] Registered new tserver with Master: d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:18:42.126120 7092 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:42.128165 6830 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.122.65:49357
I20260317 21:18:42.126619 7092 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:42.135414 7092 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:42.137012 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:42.137996 7084 heartbeater.cc:344] Connected to a master server at 127.6.122.125:32779
I20260317 21:18:42.138118 7085 heartbeater.cc:344] Connected to a master server at 127.6.122.126:40107
I20260317 21:18:42.138231 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:42.138411 7084 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:42.138450 7085 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:42.139256 7084 heartbeater.cc:507] Master 127.6.122.125:32779 requested a full tablet report, sending...
I20260317 21:18:42.139317 7085 heartbeater.cc:507] Master 127.6.122.126:40107 requested a full tablet report, sending...
I20260317 21:18:42.141004 6736 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:42.143023 7092 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:42.144719 6750 ts_manager.cc:194] Registered new tserver with Master: d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:18:42.145980 6675 ts_manager.cc:194] Registered new tserver with Master: d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:18:42.488380 7090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:42.488998 7090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:42.489681 7090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:18:42.530349 7090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:42.532361 7090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.122.66
I20260317 21:18:42.576009 7090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.122.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.122.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.122.66
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:42.577603 7090 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:42.579427 7090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:18:42.596352 7101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:42.620277 7106 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:42.623394 7106 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:42.627879 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:42.633318 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:42.636158 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:42.637604 7083 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:18:42.644933 7106 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:42.646188 7106 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:44.079347 7110 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:44.080761 7110 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:44.087082 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:44.098574 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:44.101227 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:44.109429 7110 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:44.111132 7110 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:18:43.998956 7100 debug-util.cc:398] Leaking SignalData structure 0x7b0800002660 after lost signal to thread 7090
W20260317 21:18:44.349053 7090 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.754s user 0.620s sys 1.004s
W20260317 21:18:44.350391 7090 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.755s user 0.620s sys 1.004s
W20260317 21:18:42.599586 7102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:44.351342 7103 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1753 milliseconds
W20260317 21:18:44.352620 7104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:44.352550 7090 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:18:44.357385 7090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:18:44.359802 7090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:44.361230 7090 hybrid_clock.cc:648] HybridClock initialized: now 1773782324361242 us; error 121 us; skew 500 ppm
I20260317 21:18:44.362280 7090 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:44.369792 7090 webserver.cc:492] Webserver started at http://127.6.122.66:37791/ using document root <none> and password file <none>
I20260317 21:18:44.370816 7090 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:44.371059 7090 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:44.371556 7090 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:18:44.376116 7090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data/instance:
uuid: "c0ef92fef382401a9564c17e0c223f6a"
format_stamp: "Formatted at 2026-03-17 21:18:44 on dist-test-slave-979h"
I20260317 21:18:44.377403 7090 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/instance:
uuid: "c0ef92fef382401a9564c17e0c223f6a"
format_stamp: "Formatted at 2026-03-17 21:18:44 on dist-test-slave-979h"
I20260317 21:18:44.386103 7090 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.002s
I20260317 21:18:44.392416 7116 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:44.393682 7090 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260317 21:18:44.394073 7090 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal
uuid: "c0ef92fef382401a9564c17e0c223f6a"
format_stamp: "Formatted at 2026-03-17 21:18:44 on dist-test-slave-979h"
I20260317 21:18:44.394449 7090 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:18:44.471992 7090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:18:44.473605 7090 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:18:44.474139 7090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:18:44.476817 7090 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:18:44.481969 7090 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:18:44.482259 7090 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:44.482566 7090 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:18:44.482749 7090 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:44.672176 7090 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.66:45979
I20260317 21:18:44.672292 7228 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.66:45979 every 8 connection(s)
I20260317 21:18:44.675014 7090 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/data/info.pb
I20260317 21:18:44.684314 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 7090
I20260317 21:18:44.684959 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/instance
I20260317 21:18:44.705387 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.122.67:0
--local_ip_for_outbound_sockets=127.6.122.67
--webserver_interface=127.6.122.67
--webserver_port=0
--tserver_master_addrs=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260317 21:18:44.729717 7229 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:18:44.730476 7229 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:44.732385 7229 heartbeater.cc:507] Master 127.6.122.124:38857 requested a full tablet report, sending...
I20260317 21:18:44.735833 7231 heartbeater.cc:344] Connected to a master server at 127.6.122.126:40107
I20260317 21:18:44.736095 6830 ts_manager.cc:194] Registered new tserver with Master: c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:44.736251 7231 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:44.737334 7231 heartbeater.cc:507] Master 127.6.122.126:40107 requested a full tablet report, sending...
I20260317 21:18:44.737998 6830 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.122.66:37487
I20260317 21:18:44.738932 7230 heartbeater.cc:344] Connected to a master server at 127.6.122.125:32779
I20260317 21:18:44.739286 7230 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:44.740341 6675 ts_manager.cc:194] Registered new tserver with Master: c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:44.740813 7230 heartbeater.cc:507] Master 127.6.122.125:32779 requested a full tablet report, sending...
I20260317 21:18:44.744545 6750 ts_manager.cc:194] Registered new tserver with Master: c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:45.068589 7244 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:45.069098 7244 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:45.071305 7244 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:45.072384 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:45.072513 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:45.074350 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:45.075104 7244 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:18:45.106599 7238 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:45.107177 7238 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:45.107723 7238 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:18:45.143405 7238 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:45.144840 7238 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.122.67
I20260317 21:18:45.184580 7238 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.122.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.122.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.122.67
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:45.185997 7238 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:45.187657 7238 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:18:45.202638 7246 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:45.243877 7229 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:18:45.607234 7252 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:45.608172 7252 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:45.611665 7252 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:45.616389 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:45.619423 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:45.622630 6736 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:45.624221 7252 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:45.651590 7253 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:45.652400 7253 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:45.655627 7253 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:45.657027 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:45.658360 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:45.660459 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:45.661782 7253 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:46.226032 7254 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:46.226749 7254 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:46.229351 7254 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:46.230808 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:46.231554 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:46.233552 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:46.235064 7254 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:18:45.205036 7247 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:46.596202 7249 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:46.599036 7238 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.392s user 0.000s sys 0.012s
W20260317 21:18:46.599466 7238 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.392s user 0.000s sys 0.012s
I20260317 21:18:46.599789 7238 server_base.cc:1060] running on GCE node
I20260317 21:18:46.601402 7238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:18:46.604579 7238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:46.606384 7238 hybrid_clock.cc:648] HybridClock initialized: now 1773782326606320 us; error 64 us; skew 500 ppm
I20260317 21:18:46.607645 7238 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:46.618726 7238 webserver.cc:492] Webserver started at http://127.6.122.67:37041/ using document root <none> and password file <none>
I20260317 21:18:46.620294 7238 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:46.620676 7238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:46.621403 7238 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:18:46.631636 7238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data/instance:
uuid: "2f539d1a4a584b8ca83928be49369bf4"
format_stamp: "Formatted at 2026-03-17 21:18:46 on dist-test-slave-979h"
I20260317 21:18:46.633575 7238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/instance:
uuid: "2f539d1a4a584b8ca83928be49369bf4"
format_stamp: "Formatted at 2026-03-17 21:18:46 on dist-test-slave-979h"
I20260317 21:18:46.646274 7238 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.007s sys 0.005s
I20260317 21:18:46.656014 7260 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:46.657775 7238 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260317 21:18:46.658277 7238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal
uuid: "2f539d1a4a584b8ca83928be49369bf4"
format_stamp: "Formatted at 2026-03-17 21:18:46 on dist-test-slave-979h"
I20260317 21:18:46.658772 7238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:18:46.760108 7238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:18:46.761868 7238 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:18:46.762398 7238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:18:46.769200 7238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:18:46.774050 7238 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:18:46.774278 7238 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:46.774487 7238 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:18:46.774624 7238 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:18:46.787070 7269 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:46.787525 7269 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:46.789789 7269 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:46.790963 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:46.790962 6840 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:18:46.792773 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:46.793773 7269 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:46.980810 7238 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.67:32877
I20260317 21:18:46.980950 7373 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.67:32877 every 8 connection(s)
I20260317 21:18:46.984895 7238 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/data/info.pb
I20260317 21:18:46.988307 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 7238
I20260317 21:18:46.988957 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/instance
I20260317 21:18:47.035168 7374 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:18:47.035694 7374 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:47.037653 7374 heartbeater.cc:507] Master 127.6.122.124:38857 requested a full tablet report, sending...
I20260317 21:18:47.038552 7375 heartbeater.cc:344] Connected to a master server at 127.6.122.125:32779
I20260317 21:18:47.038599 7377 heartbeater.cc:344] Connected to a master server at 127.6.122.126:40107
I20260317 21:18:47.038977 7375 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:47.038977 7377 heartbeater.cc:461] Registering TS with master...
I20260317 21:18:47.040313 7377 heartbeater.cc:507] Master 127.6.122.126:40107 requested a full tablet report, sending...
I20260317 21:18:47.040321 7375 heartbeater.cc:507] Master 127.6.122.125:32779 requested a full tablet report, sending...
I20260317 21:18:47.040774 6830 ts_manager.cc:194] Registered new tserver with Master: 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:47.043332 6830 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.122.67:51023
I20260317 21:18:47.043509 6675 ts_manager.cc:194] Registered new tserver with Master: 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:47.043735 6750 ts_manager.cc:194] Registered new tserver with Master: 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:47.068508 6633 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260317 21:18:47.149956 6830 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48742:
name: "default.Table_0c99f90b436949639fbdb78a85d694c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:47.152762 6829 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48742:
name: "default.Table_93d3d7227ef94de4aed8fb5b404439dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:47.153973 6830 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c99f90b436949639fbdb78a85d694c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:47.154884 6829 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93d3d7227ef94de4aed8fb5b404439dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:47.154914 6827 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48742:
name: "default.Table_3d620e0da0074411b2996015acc9f0b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:47.156251 6828 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:48742:
name: "default.Table_ac0ee103335146a3b4168471de3fd782"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:47.160079 6828 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac0ee103335146a3b4168471de3fd782 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:47.160006 6827 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d620e0da0074411b2996015acc9f0b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:47.225304 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 6787
W20260317 21:18:47.255676 6974 connection.cc:537] client connection to 127.6.122.124:38857 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:18:47.256572 7083 heartbeater.cc:646] Failed to heartbeat to 127.6.122.124:38857 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:18:47.257434 6734 connection.cc:537] server connection from 127.0.0.1:56982 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:18:47.275683 7229 heartbeater.cc:646] Failed to heartbeat to 127.6.122.124:38857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
W20260317 21:18:47.530613 7079 debug-util.cc:398] Leaking SignalData structure 0x7b08000b4160 after lost signal to thread 6941
I20260317 21:18:47.547021 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
W20260317 21:18:47.556869 7374 heartbeater.cc:646] Failed to heartbeat to 127.6.122.124:38857 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:47.701972 7418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:47.702890 7418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:47.706282 7418 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:47.708385 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:18:47.709179 6759 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31726fd7f40442e49b2fe0f880793206 in term 1.
I20260317 21:18:47.710804 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206; no voters:
I20260317 21:18:47.714921 7417 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:47.716337 7417 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:47.718196 7418 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 21:18:47.718705 7418 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 47e1c983d0224428a9a7b147a0e5e4ef)
I20260317 21:18:47.719779 7418 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:18:47.727622 7417 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
W20260317 21:18:47.748296 6737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111) [suppressed 6 similar messages]
W20260317 21:18:47.758582 6662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111) [suppressed 13 similar messages]
I20260317 21:18:47.765767 7418 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
W20260317 21:18:47.775098 6737 leader_election.cc:336] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:47.788926 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:18:47.789562 6759 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:18:47.803661 7418 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 election: Requested vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
W20260317 21:18:47.808682 6662 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 50106.4us
I20260317 21:18:47.811450 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:18:47.812389 6683 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 21186091f2c9410ea6e585c5ef2c6291 in current term 2: Already voted for candidate 31726fd7f40442e49b2fe0f880793206 in this term.
I20260317 21:18:47.815853 6736 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
W20260317 21:18:47.815516 6662 leader_election.cc:336] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
W20260317 21:18:47.822772 6662 leader_election.cc:336] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:47.868538 6759 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31726fd7f40442e49b2fe0f880793206 in term 2.
I20260317 21:18:47.870550 7417 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:47.875933 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206; no voters: 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:47.880553 7418 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:18:47.883685 7418 raft_consensus.cc:697] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 LEADER]: Becoming Leader. State: Replica: 31726fd7f40442e49b2fe0f880793206, State: Running, Role: LEADER
I20260317 21:18:47.885003 7418 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:47.890600 7425 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31726fd7f40442e49b2fe0f880793206. Latest consensus state: current_term: 2 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:47.891892 7425 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: LEADER
I20260317 21:18:48.375948 7417 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:48.376474 7417 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:18:48.378742 7417 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:18:48.381158 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 3 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
W20260317 21:18:48.389772 6737 leader_election.cc:336] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111)
I20260317 21:18:48.390352 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291; no voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:18:48.391407 7417 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 21:18:48.433087 6759 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Refusing update from remote peer 31726fd7f40442e49b2fe0f880793206: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260317 21:18:48.435097 7432 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Connected to new peer: Peer: permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 4, Time since last communication: 0.000s
I20260317 21:18:48.440347 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31726fd7f40442e49b2fe0f880793206. Latest consensus state: current_term: 2 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:48.441241 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:48.475389 7417 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:48.476258 7417 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:18:48.480540 7432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:18:48.481431 7432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: LEADER
W20260317 21:18:48.486776 6662 consensus_peers.cc:597] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 -> Peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Couldn't send request to peer 47e1c983d0224428a9a7b147a0e5e4ef. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:48.564126 7426 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:18:48.574740 7426 catalog_manager.cc:679] Loaded metadata for table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]
I20260317 21:18:48.583400 7426 tablet_loader.cc:96] loaded metadata for tablet a143e0f0100143e2bae1423d5d1be4b5 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2])
I20260317 21:18:48.585165 7426 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:18:48.590747 7426 catalog_manager.cc:1269] Loaded cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:18:48.591187 7426 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:18:48.596956 7426 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:18:48.601972 7426 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Loaded TSK: 0
I20260317 21:18:48.604558 7426 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20260317 21:18:48.559183 6638 debug-util.cc:398] Leaking SignalData structure 0x7b080002d0a0 after lost signal to thread 7387
W20260317 21:18:47.257231 7387 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.124:38857)
W20260317 21:18:48.788352 6638 kernel_stack_watchdog.cc:198] Thread 7387 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 301ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260317 21:18:48.798321 7387 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.CreateTable -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 1.54033e+06us
I20260317 21:18:48.799080 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:38857
--webserver_interface=127.6.122.124
--webserver_port=40359
--builtin_ntp_servers=127.6.122.84:43297
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260317 21:18:48.800235 7387 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.124:38857)
I20260317 21:18:48.843168 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_ac0ee103335146a3b4168471de3fd782"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:48.842882 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0c99f90b436949639fbdb78a85d694c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:48.841823 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3d620e0da0074411b2996015acc9f0b6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:48.842443 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_93d3d7227ef94de4aed8fb5b404439dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:48.877872 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c99f90b436949639fbdb78a85d694c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:48.878271 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93d3d7227ef94de4aed8fb5b404439dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:48.880053 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac0ee103335146a3b4168471de3fd782 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:48.882375 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d620e0da0074411b2996015acc9f0b6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:48.904278 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d1a9cf61408840c19f65bb910a1b56f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:48.906292 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1a9cf61408840c19f65bb910a1b56f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:48.942394 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0c99f90b436949639fbdb78a85d694c9" } new_table_name: "default.Table_d3b3ef2c53324b2ba62c7440297657bb" modify_external_catalogs: true
I20260317 21:18:48.977676 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_3d620e0da0074411b2996015acc9f0b6" } new_table_name: "default.Table_55f3cb3f8e6a4dd8b02827ec8310ba7c" modify_external_catalogs: true
I20260317 21:18:48.989753 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_93d3d7227ef94de4aed8fb5b404439dc" } modify_external_catalogs: true
I20260317 21:18:48.990437 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_93d3d7227ef94de4aed8fb5b404439dc" } modify_external_catalogs: true
I20260317 21:18:49.054196 6674 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d1a9cf61408840c19f65bb910a1b56f5" } modify_external_catalogs: true
I20260317 21:18:49.055012 6674 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d1a9cf61408840c19f65bb910a1b56f5" } modify_external_catalogs: true
W20260317 21:18:49.109294 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3255f88404e5407ab3663a3a52b71730 consensus state has no leader
I20260317 21:18:49.119511 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f6135762cd8b4b4dafa699721b7bf703"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.121742 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6135762cd8b4b4dafa699721b7bf703 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.131575 7163 tablet_service.cc:1511] Processing CreateTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.134509 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a143e0f0100143e2bae1423d5d1be4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.141148 7164 tablet_service.cc:1511] Processing CreateTablet for tablet 3255f88404e5407ab3663a3a52b71730 (DEFAULT_TABLE table=default.Table_0c99f90b436949639fbdb78a85d694c9 [id=cc8571e1fc7a4721a53d0deab165acc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.141147 7018 tablet_service.cc:1511] Processing CreateTablet for tablet 3255f88404e5407ab3663a3a52b71730 (DEFAULT_TABLE table=default.Table_0c99f90b436949639fbdb78a85d694c9 [id=cc8571e1fc7a4721a53d0deab165acc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.144384 7017 tablet_service.cc:1511] Processing CreateTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.147869 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a143e0f0100143e2bae1423d5d1be4b5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:49.150432 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00e7af4aa99341189d3001c9188689d7 has no consensus state
I20260317 21:18:49.157687 7018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3255f88404e5407ab3663a3a52b71730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.177378 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_5fce106f6b6a4e00b9af07d5e5fc930f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.179489 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:49.180502 6662 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 3255f88404e5407ab3663a3a52b71730 (table default.Table_d3b3ef2c53324b2ba62c7440297657bb [id=cc8571e1fc7a4721a53d0deab165acc0], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 3255f88404e5407ab3663a3a52b71730 consensus state has no leader
I20260317 21:18:49.179158 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_6cf439fbd1a94717982b254fa652fabd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.186892 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6cf439fbd1a94717982b254fa652fabd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.189158 6675 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet ae5ce3dc5df94dc0a33e7baabfbce8cc
I20260317 21:18:49.199432 6674 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 5e445e7adc5e45f88124a219e15301ab
I20260317 21:18:49.217615 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3255f88404e5407ab3663a3a52b71730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.217478 7308 tablet_service.cc:1511] Processing CreateTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.219099 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 3255f88404e5407ab3663a3a52b71730 (DEFAULT_TABLE table=default.Table_0c99f90b436949639fbdb78a85d694c9 [id=cc8571e1fc7a4721a53d0deab165acc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.220908 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3255f88404e5407ab3663a3a52b71730. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.223224 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_844e7186ad624a5da968ffef9aac1610"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.225164 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_844e7186ad624a5da968ffef9aac1610 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.225679 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a143e0f0100143e2bae1423d5d1be4b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.282148 7459 tablet_bootstrap.cc:492] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:49.282332 7085 heartbeater.cc:499] Master 127.6.122.126:40107 was elected leader, sending a full tablet report...
I20260317 21:18:49.298239 7231 heartbeater.cc:499] Master 127.6.122.126:40107 was elected leader, sending a full tablet report...
I20260317 21:18:49.302817 7459 tablet_bootstrap.cc:654] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.305956 7459 log.cc:826] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Log is configured to *not* fsync() on all Append() calls
I20260317 21:18:49.312564 7462 tablet_bootstrap.cc:492] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:49.316190 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet a143e0f0100143e2bae1423d5d1be4b5 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:49.321228 6675 master_service.cc:954] ReplaceTablet: received request to replace tablet 3255f88404e5407ab3663a3a52b71730 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:49.327394 7459 tablet_bootstrap.cc:492] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:49.328245 7459 ts_tablet_manager.cc:1403] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.007s
I20260317 21:18:49.336354 6670 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6cf439fbd1a94717982b254fa652fabd" } new_table_name: "default.Table_562f2136d87742b4839f28ff31c05f40" modify_external_catalogs: true
I20260317 21:18:49.341467 7462 tablet_bootstrap.cc:654] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.344422 7462 log.cc:826] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Log is configured to *not* fsync() on all Append() calls
I20260317 21:18:49.355698 7018 tablet_service.cc:1511] Processing CreateTablet for tablet 00e7af4aa99341189d3001c9188689d7 (DEFAULT_TABLE table=default.Table_55f3cb3f8e6a4dd8b02827ec8310ba7c [id=ccab17f886b74e94a13a9ad95275089a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.356989 7018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e7af4aa99341189d3001c9188689d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.356976 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 00e7af4aa99341189d3001c9188689d7 (DEFAULT_TABLE table=default.Table_55f3cb3f8e6a4dd8b02827ec8310ba7c [id=ccab17f886b74e94a13a9ad95275089a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.358223 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e7af4aa99341189d3001c9188689d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.380029 7462 tablet_bootstrap.cc:492] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:49.380875 7462 ts_tablet_manager.cc:1403] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.069s user 0.019s sys 0.008s
I20260317 21:18:49.382844 7459 raft_consensus.cc:359] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.384373 7459 raft_consensus.cc:385] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.381752 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 00e7af4aa99341189d3001c9188689d7 (DEFAULT_TABLE table=default.Table_55f3cb3f8e6a4dd8b02827ec8310ba7c [id=ccab17f886b74e94a13a9ad95275089a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.384765 7459 raft_consensus.cc:740] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.385633 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00e7af4aa99341189d3001c9188689d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.389395 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_55f3cb3f8e6a4dd8b02827ec8310ba7c" } new_table_name: "default.Table_12f97907f120466093dbd44c0cd929aa" modify_external_catalogs: true
I20260317 21:18:49.392408 7459 consensus_queue.cc:260] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.402124 7467 tablet_bootstrap.cc:492] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:49.409390 7459 ts_tablet_manager.cc:1434] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.081s user 0.034s sys 0.006s
I20260317 21:18:49.409235 6673 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d3b3ef2c53324b2ba62c7440297657bb" } modify_external_catalogs: true
I20260317 21:18:49.409865 6673 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d3b3ef2c53324b2ba62c7440297657bb" } modify_external_catalogs: true
I20260317 21:18:49.414033 6669 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f6135762cd8b4b4dafa699721b7bf703" } modify_external_catalogs: true
I20260317 21:18:49.414608 6669 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f6135762cd8b4b4dafa699721b7bf703" } modify_external_catalogs: true
I20260317 21:18:49.415105 7459 tablet_bootstrap.cc:492] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:49.417835 7467 tablet_bootstrap.cc:654] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.425560 7467 log.cc:826] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Log is configured to *not* fsync() on all Append() calls
I20260317 21:18:49.432179 7459 tablet_bootstrap.cc:654] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.445252 7462 raft_consensus.cc:359] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.446555 7462 raft_consensus.cc:385] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.446956 7462 raft_consensus.cc:740] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.447518 7467 tablet_bootstrap.cc:492] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:49.448158 7467 ts_tablet_manager.cc:1403] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.047s user 0.015s sys 0.006s
I20260317 21:18:49.448149 7462 consensus_queue.cc:260] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.454864 7459 tablet_bootstrap.cc:492] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:49.455246 7462 ts_tablet_manager.cc:1434] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.073s user 0.031s sys 0.011s
I20260317 21:18:49.455418 7459 ts_tablet_manager.cc:1403] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.041s user 0.006s sys 0.007s
I20260317 21:18:49.459254 7462 tablet_bootstrap.cc:492] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:49.460933 7377 heartbeater.cc:499] Master 127.6.122.126:40107 was elected leader, sending a full tablet report...
I20260317 21:18:49.458463 7459 raft_consensus.cc:359] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.462862 7459 raft_consensus.cc:385] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.463279 7459 raft_consensus.cc:740] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.464048 7459 consensus_queue.cc:260] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.469079 7459 ts_tablet_manager.cc:1434] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.013s user 0.004s sys 0.003s
I20260317 21:18:49.472064 7459 tablet_bootstrap.cc:492] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:49.473706 7462 tablet_bootstrap.cc:654] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.479126 7459 tablet_bootstrap.cc:654] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.492231 7459 tablet_bootstrap.cc:492] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:49.492995 7459 ts_tablet_manager.cc:1403] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.021s user 0.017s sys 0.002s
I20260317 21:18:49.493166 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f3adbc8bf1994597bf7be6a6792ebf5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.495132 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3adbc8bf1994597bf7be6a6792ebf5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.496119 7459 raft_consensus.cc:359] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.497148 7459 raft_consensus.cc:385] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.497589 7459 raft_consensus.cc:740] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.498871 7459 consensus_queue.cc:260] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.502306 7459 ts_tablet_manager.cc:1434] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260317 21:18:49.504892 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_857e967088644fd98151128bb9ebdd4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.507113 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_857e967088644fd98151128bb9ebdd4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.514572 7462 tablet_bootstrap.cc:492] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:49.515058 7462 ts_tablet_manager.cc:1403] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.009s
I20260317 21:18:49.516809 7467 raft_consensus.cc:359] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.517908 7467 raft_consensus.cc:385] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.518227 7467 raft_consensus.cc:740] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.518383 7462 raft_consensus.cc:359] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:49.519140 7379 tablet.cc:2379] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:18:49.519285 7462 raft_consensus.cc:385] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.519202 7467 consensus_queue.cc:260] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.519716 7462 raft_consensus.cc:740] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.520555 7462 consensus_queue.cc:260] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:49.525413 6670 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 31725534b2a9401e9a79e8a467ed5210 has no consensus state
I20260317 21:18:49.528935 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 3255f88404e5407ab3663a3a52b71730
I20260317 21:18:49.530766 6675 catalog_manager.cc:6741] ReplaceTablet: tablet 3255f88404e5407ab3663a3a52b71730 deleted and replaced by tablet 1ab2868e51e74828bdd3d68ddbb0477e
I20260317 21:18:49.531487 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet a143e0f0100143e2bae1423d5d1be4b5
I20260317 21:18:49.532974 6672 catalog_manager.cc:6741] ReplaceTablet: tablet a143e0f0100143e2bae1423d5d1be4b5 deleted and replaced by tablet b42fa238590b47d5bc0bd43dd3a313f2
I20260317 21:18:49.532958 7467 ts_tablet_manager.cc:1434] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.084s user 0.023s sys 0.014s
I20260317 21:18:49.532588 7462 ts_tablet_manager.cc:1434] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260317 21:18:49.537302 7018 tablet_service.cc:1558] Processing DeleteTablet for tablet 3255f88404e5407ab3663a3a52b71730 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ab2868e51e74828bdd3d68ddbb0477e) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:49.539460 7462 tablet_bootstrap.cc:492] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:49.542110 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 3255f88404e5407ab3663a3a52b71730 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ab2868e51e74828bdd3d68ddbb0477e) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:49.545680 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b42fa238590b47d5bc0bd43dd3a313f2) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:49.549744 7468 raft_consensus.cc:493] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:49.550559 7468 raft_consensus.cc:515] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.552762 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b42fa238590b47d5bc0bd43dd3a313f2) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:49.561044 7468 leader_election.cc:290] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:49.588896 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00e7af4aa99341189d3001c9188689d7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:49.590184 7091 tablet.cc:2379] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260317 21:18:49.592257 6971 leader_election.cc:343] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:49.592837 7467 tablet_bootstrap.cc:492] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:49.596726 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet 3255f88404e5407ab3663a3a52b71730 with delete_type TABLET_DATA_DELETED (replaced by tablet 1ab2868e51e74828bdd3d68ddbb0477e) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:49.598919 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b42fa238590b47d5bc0bd43dd3a313f2) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:49.604460 7477 tablet_replica.cc:333] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:49.605767 7477 raft_consensus.cc:2243] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.607421 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4909c6d723f84e1d90b90e361104e061"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.610106 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4909c6d723f84e1d90b90e361104e061 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.613996 7462 tablet_bootstrap.cc:654] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.614374 7467 tablet_bootstrap.cc:654] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.615746 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_5c0d07d92b2d4bd3bfefa18fc6803f7a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.617700 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5c0d07d92b2d4bd3bfefa18fc6803f7a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.614723 7478 tablet_replica.cc:333] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:49.620795 7478 raft_consensus.cc:2243] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.621678 7478 raft_consensus.cc:2272] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.621973 7477 raft_consensus.cc:2272] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.626446 7478 ts_tablet_manager.cc:1916] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:49.626621 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00e7af4aa99341189d3001c9188689d7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:18:49.629441 6973 leader_election.cc:343] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:49.630735 7477 ts_tablet_manager.cc:1916] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:49.630681 6973 leader_election.cc:304] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:49.633965 7468 raft_consensus.cc:2749] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:49.636530 7467 tablet_bootstrap.cc:492] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:49.637266 7467 ts_tablet_manager.cc:1403] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.102s user 0.017s sys 0.016s
I20260317 21:18:49.647094 7462 tablet_bootstrap.cc:492] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:49.649075 7462 ts_tablet_manager.cc:1403] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.110s user 0.013s sys 0.005s
I20260317 21:18:49.652308 7462 raft_consensus.cc:359] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.641749 7467 raft_consensus.cc:359] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.654310 7467 raft_consensus.cc:385] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.654799 7467 raft_consensus.cc:740] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.655776 7462 raft_consensus.cc:385] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.655897 7467 consensus_queue.cc:260] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.656631 7462 raft_consensus.cc:740] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.658022 7462 consensus_queue.cc:260] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.659180 7467 ts_tablet_manager.cc:1434] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.021s user 0.008s sys 0.001s
I20260317 21:18:49.662981 7462 ts_tablet_manager.cc:1434] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260317 21:18:49.660431 7467 tablet_bootstrap.cc:492] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:49.669976 7478 ts_tablet_manager.cc:1929] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:49.654851 7483 tablet_replica.cc:333] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:49.673545 7477 ts_tablet_manager.cc:1929] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:49.674258 7478 log.cc:1199] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/3255f88404e5407ab3663a3a52b71730
I20260317 21:18:49.674760 7477 log.cc:1199] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/3255f88404e5407ab3663a3a52b71730
I20260317 21:18:49.675056 7483 raft_consensus.cc:2243] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.677454 7477 ts_tablet_manager.cc:1950] T 3255f88404e5407ab3663a3a52b71730 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:49.676589 7478 ts_tablet_manager.cc:1950] T 3255f88404e5407ab3663a3a52b71730 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:49.684368 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 3255f88404e5407ab3663a3a52b71730 (table default.Table_d3b3ef2c53324b2ba62c7440297657bb [id=cc8571e1fc7a4721a53d0deab165acc0]) successfully deleted
I20260317 21:18:49.687299 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 3255f88404e5407ab3663a3a52b71730 (table default.Table_d3b3ef2c53324b2ba62c7440297657bb [id=cc8571e1fc7a4721a53d0deab165acc0]) successfully deleted
I20260317 21:18:49.689766 7483 raft_consensus.cc:2272] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.696080 6669 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 50532643be96404c805bca853fa38a4d
I20260317 21:18:49.698096 7477 tablet_replica.cc:333] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:49.699065 7477 raft_consensus.cc:2243] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.699599 7477 raft_consensus.cc:2272] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.700608 7478 tablet_replica.cc:333] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:49.701324 7478 raft_consensus.cc:2243] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.701828 7478 raft_consensus.cc:2272] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.704107 7477 ts_tablet_manager.cc:1916] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:49.708874 7483 ts_tablet_manager.cc:1916] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:49.713128 7467 tablet_bootstrap.cc:654] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.714359 7478 ts_tablet_manager.cc:1916] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:49.748373 7467 tablet_bootstrap.cc:492] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:49.750362 7467 ts_tablet_manager.cc:1403] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.090s user 0.011s sys 0.007s
I20260317 21:18:49.750449 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_857e967088644fd98151128bb9ebdd4b" } new_table_name: "default.Table_66e3f82e56254582a35f66ba0e24ac57" modify_external_catalogs: true
I20260317 21:18:49.755244 7483 ts_tablet_manager.cc:1929] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:49.756518 7483 log.cc:1199] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/3255f88404e5407ab3663a3a52b71730
I20260317 21:18:49.757531 7483 ts_tablet_manager.cc:1950] T 3255f88404e5407ab3663a3a52b71730 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:49.761044 7483 tablet_replica.cc:333] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
W20260317 21:18:49.761287 7443 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:18:49.762094 7443 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:18:49.762848 7443 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260317 21:18:49.763085 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 3255f88404e5407ab3663a3a52b71730 (table default.Table_d3b3ef2c53324b2ba62c7440297657bb [id=cc8571e1fc7a4721a53d0deab165acc0]) successfully deleted
I20260317 21:18:49.766810 7467 raft_consensus.cc:359] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:49.768827 7467 raft_consensus.cc:385] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:49.769809 7483 raft_consensus.cc:2243] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.770365 7483 raft_consensus.cc:2272] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.773108 6670 master_service.cc:954] ReplaceTablet: received request to replace tablet 00e7af4aa99341189d3001c9188689d7 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:49.773670 6669 master_service.cc:954] ReplaceTablet: received request to replace tablet 00e7af4aa99341189d3001c9188689d7 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:49.770259 7467 raft_consensus.cc:740] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:49.775846 7483 ts_tablet_manager.cc:1916] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:49.775596 7467 consensus_queue.cc:260] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:49.781162 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 742b03ea3048466da3c9ea5ed9010b27 has no consensus state
I20260317 21:18:49.789407 7467 ts_tablet_manager.cc:1434] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.039s user 0.001s sys 0.008s
I20260317 21:18:49.809924 7483 ts_tablet_manager.cc:1929] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:49.810364 7483 log.cc:1199] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/a143e0f0100143e2bae1423d5d1be4b5
I20260317 21:18:49.811160 7483 ts_tablet_manager.cc:1950] T a143e0f0100143e2bae1423d5d1be4b5 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:49.814703 7477 ts_tablet_manager.cc:1929] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:49.815124 7477 log.cc:1199] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/a143e0f0100143e2bae1423d5d1be4b5
I20260317 21:18:49.815783 7477 ts_tablet_manager.cc:1950] T a143e0f0100143e2bae1423d5d1be4b5 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:49.815800 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet a143e0f0100143e2bae1423d5d1be4b5 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:18:49.825870 7478 ts_tablet_manager.cc:1929] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:49.827171 7478 log.cc:1199] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/a143e0f0100143e2bae1423d5d1be4b5
I20260317 21:18:49.827513 6668 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 00e7af4aa99341189d3001c9188689d7 (table default.Table_55f3cb3f8e6a4dd8b02827ec8310ba7c [id=ccab17f886b74e94a13a9ad95275089a]). Expected version 1 got 0
I20260317 21:18:49.828058 7478 ts_tablet_manager.cc:1950] T a143e0f0100143e2bae1423d5d1be4b5 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:49.828316 7018 tablet_service.cc:1511] Processing CreateTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 (DEFAULT_TABLE table=default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.830220 7018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc0fe1bcaeb04f7eb157172f9c216d56. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:49.832468 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00e7af4aa99341189d3001c9188689d7 consensus state has no leader
I20260317 21:18:49.833496 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet a143e0f0100143e2bae1423d5d1be4b5 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:18:49.834550 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet a143e0f0100143e2bae1423d5d1be4b5 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:18:49.840855 6673 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 1ab2868e51e74828bdd3d68ddbb0477e
I20260317 21:18:49.849603 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 3255f88404e5407ab3663a3a52b71730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:49 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:49.856630 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 3255f88404e5407ab3663a3a52b71730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:49 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:49.856853 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b42fa238590b47d5bc0bd43dd3a313f2) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:49.857813 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 3255f88404e5407ab3663a3a52b71730 because the tablet was not found. No further retry: Not found: Tablet not found: 3255f88404e5407ab3663a3a52b71730
W20260317 21:18:49.859701 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 3255f88404e5407ab3663a3a52b71730 because the tablet was not found. No further retry: Not found: Tablet not found: 3255f88404e5407ab3663a3a52b71730
W20260317 21:18:49.860591 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet a143e0f0100143e2bae1423d5d1be4b5 because the tablet was not found. No further retry: Not found: Tablet not found: a143e0f0100143e2bae1423d5d1be4b5
I20260317 21:18:49.864219 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet a143e0f0100143e2bae1423d5d1be4b5 with delete_type TABLET_DATA_DELETED (replaced by tablet b42fa238590b47d5bc0bd43dd3a313f2) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:49.868772 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet 3255f88404e5407ab3663a3a52b71730 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:49 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:49.870850 7443 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:18:49.871315 7443 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:18:49.871753 7443 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:18:49.872112 7443 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:18:49.877828 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 31725534b2a9401e9a79e8a467ed5210 (DEFAULT_TABLE table=default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.878523 7162 tablet_service.cc:1511] Processing CreateTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 (DEFAULT_TABLE table=default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.879181 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31725534b2a9401e9a79e8a467ed5210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.879632 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc0fe1bcaeb04f7eb157172f9c216d56. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.892238 7308 tablet_service.cc:1511] Processing CreateTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 (DEFAULT_TABLE table=default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.893728 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc0fe1bcaeb04f7eb157172f9c216d56. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:49.909194 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet a143e0f0100143e2bae1423d5d1be4b5 because the tablet was not found. No further retry: Not found: Tablet not found: a143e0f0100143e2bae1423d5d1be4b5
I20260317 21:18:49.914147 7459 tablet_bootstrap.cc:492] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:49.918231 7164 tablet_service.cc:1511] Processing CreateTablet for tablet 31725534b2a9401e9a79e8a467ed5210 (DEFAULT_TABLE table=default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.919539 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31725534b2a9401e9a79e8a467ed5210. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:49.914997 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 3255f88404e5407ab3663a3a52b71730 because the tablet was not found. No further retry: Not found: Tablet not found: 3255f88404e5407ab3663a3a52b71730
I20260317 21:18:49.922367 7459 tablet_bootstrap.cc:654] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:49.933380 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 31725534b2a9401e9a79e8a467ed5210 (DEFAULT_TABLE table=default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.934808 7475 raft_consensus.cc:493] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:49.934780 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a7b4fa95d87c4a10895ca301321c5738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:49.935679 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31725534b2a9401e9a79e8a467ed5210. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.935398 7475 raft_consensus.cc:515] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:49.937223 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7b4fa95d87c4a10895ca301321c5738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.948891 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 (DEFAULT_TABLE table=default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.949410 7475 leader_election.cc:290] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:49.950449 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a3f11d9d4d4037af60cf1a9b4f3377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.950361 7163 tablet_service.cc:1511] Processing CreateTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 (DEFAULT_TABLE table=default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:49.951526 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a3f11d9d4d4037af60cf1a9b4f3377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:49.952174 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_7a6911ff81c44dbf87a617dd4ceeb1e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.957834 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a6911ff81c44dbf87a617dd4ceeb1e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.967578 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_9f87958031d24bfbb6e275b81361fc4e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:49.964222 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_12f97907f120466093dbd44c0cd929aa" } modify_external_catalogs: true
I20260317 21:18:49.972028 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_12f97907f120466093dbd44c0cd929aa" } modify_external_catalogs: true
I20260317 21:18:49.971271 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_5bd5abc0a4a64c408b1bae33441413d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:49.977223 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bd5abc0a4a64c408b1bae33441413d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:49.981613 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f87958031d24bfbb6e275b81361fc4e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:49.981946 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 00e7af4aa99341189d3001c9188689d7
I20260317 21:18:49.983444 6670 catalog_manager.cc:6741] ReplaceTablet: tablet 00e7af4aa99341189d3001c9188689d7 deleted and replaced by tablet 6343bd5fc6a44ac3976cbbed8fab26c7
I20260317 21:18:49.983716 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 00e7af4aa99341189d3001c9188689d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6343bd5fc6a44ac3976cbbed8fab26c7) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:49.984496 7477 tablet_replica.cc:333] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:18:49.984789 7239 tablet.cc:2379] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:18:49.989077 7161 tablet_service.cc:1558] Processing DeleteTablet for tablet 00e7af4aa99341189d3001c9188689d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6343bd5fc6a44ac3976cbbed8fab26c7) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:49.989938 7478 tablet_replica.cc:333] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:49.990661 7478 raft_consensus.cc:2243] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:49.991312 7478 raft_consensus.cc:2272] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:49.996398 7478 ts_tablet_manager.cc:1916] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:50.007726 7477 raft_consensus.cc:2243] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:50.008409 7477 raft_consensus.cc:2272] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:50.010524 7477 ts_tablet_manager.cc:1916] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:50.012166 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00e7af4aa99341189d3001c9188689d7" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:50.013273 7329 raft_consensus.cc:2468] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:50.018959 7467 tablet_bootstrap.cc:492] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:50.014956 7119 leader_election.cc:304] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
W20260317 21:18:50.020054 7119 raft_consensus.cc:2686] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:18:50.027002 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 00e7af4aa99341189d3001c9188689d7 with delete_type TABLET_DATA_DELETED (replaced by tablet 6343bd5fc6a44ac3976cbbed8fab26c7) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:50.027930 7483 tablet_replica.cc:333] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:50.029503 7483 raft_consensus.cc:2243] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:50.030092 7483 raft_consensus.cc:2272] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:50.033996 7483 ts_tablet_manager.cc:1916] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:50.034222 7467 tablet_bootstrap.cc:654] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.066990 7459 tablet_bootstrap.cc:492] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:50.067736 7459 ts_tablet_manager.cc:1403] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.154s user 0.012s sys 0.000s
I20260317 21:18:50.070947 7459 raft_consensus.cc:359] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.081797 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_66e3f82e56254582a35f66ba0e24ac57" } new_table_name: "default.Table_488efb7758fb4a7a8bdbfae0f0864951" modify_external_catalogs: true
I20260317 21:18:50.084515 7467 tablet_bootstrap.cc:492] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:50.085256 7467 ts_tablet_manager.cc:1403] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.067s user 0.020s sys 0.011s
I20260317 21:18:50.091954 7307 tablet_service.cc:1511] Processing CreateTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 (DEFAULT_TABLE table=default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.093533 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 87a3f11d9d4d4037af60cf1a9b4f3377. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.093412 7467 raft_consensus.cc:359] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.094154 7467 raft_consensus.cc:385] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.094460 7467 raft_consensus.cc:740] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.095252 7467 consensus_queue.cc:260] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.096591 7443 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:43297
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:40107,127.6.122.125:32779,127.6.122.124:38857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:38857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.122.124
--webserver_port=40359
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:18:50.097698 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4909c6d723f84e1d90b90e361104e061" } new_table_name: "default.Table_f671acaa7a4b41fcb385266ab7e21141" modify_external_catalogs: true
I20260317 21:18:50.098431 7443 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:18:50.104128 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f3adbc8bf1994597bf7be6a6792ebf5a" } modify_external_catalogs: true
I20260317 21:18:50.104875 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f3adbc8bf1994597bf7be6a6792ebf5a" } modify_external_catalogs: true
I20260317 21:18:50.120898 7459 raft_consensus.cc:385] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.121328 7459 raft_consensus.cc:740] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.121662 7443 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260317 21:18:50.122592 7459 consensus_queue.cc:260] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.104516 7478 ts_tablet_manager.cc:1929] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:50.125679 7478 log.cc:1199] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/00e7af4aa99341189d3001c9188689d7
I20260317 21:18:50.119683 7467 ts_tablet_manager.cc:1434] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.034s user 0.002s sys 0.006s
I20260317 21:18:50.127034 7462 tablet_bootstrap.cc:492] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:50.134825 7462 tablet_bootstrap.cc:654] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.136746 7467 tablet_bootstrap.cc:492] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:50.138760 7483 ts_tablet_manager.cc:1929] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:50.139212 7483 log.cc:1199] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/00e7af4aa99341189d3001c9188689d7
I20260317 21:18:50.140146 7483 ts_tablet_manager.cc:1950] T 00e7af4aa99341189d3001c9188689d7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:50.140350 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00e7af4aa99341189d3001c9188689d7" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:50.143276 7120 leader_election.cc:343] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:50.154559 7459 ts_tablet_manager.cc:1434] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.086s user 0.008s sys 0.001s
I20260317 21:18:50.155587 7475 raft_consensus.cc:493] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:50.156049 7475 raft_consensus.cc:515] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.158353 7475 leader_election.cc:290] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:50.161818 6675 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 6343bd5fc6a44ac3976cbbed8fab26c7
W20260317 21:18:50.161922 7119 leader_election.cc:343] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 31725534b2a9401e9a79e8a467ed5210
I20260317 21:18:50.165167 7467 tablet_bootstrap.cc:654] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.170928 7478 ts_tablet_manager.cc:1950] T 00e7af4aa99341189d3001c9188689d7 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
W20260317 21:18:50.180962 6660 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 31725534b2a9401e9a79e8a467ed5210 consensus state has no leader
I20260317 21:18:50.214164 7462 tablet_bootstrap.cc:492] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.215216 7462 ts_tablet_manager.cc:1403] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.088s user 0.011s sys 0.004s
I20260317 21:18:50.219580 7462 raft_consensus.cc:359] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.220839 7462 raft_consensus.cc:385] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.221194 7462 raft_consensus.cc:740] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.222214 7462 consensus_queue.cc:260] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.163959 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31725534b2a9401e9a79e8a467ed5210" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:50.225122 7038 raft_consensus.cc:2468] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:50.226966 7120 leader_election.cc:304] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:50.228636 7475 raft_consensus.cc:2804] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:50.229749 7475 raft_consensus.cc:493] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:50.230604 7475 raft_consensus.cc:3060] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:50.255887 6668 master_service.cc:954] ReplaceTablet: received request to replace tablet 87a3f11d9d4d4037af60cf1a9b4f3377 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:50.259608 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet fc0fe1bcaeb04f7eb157172f9c216d56 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:50.263623 7459 tablet_bootstrap.cc:492] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:18:50.294523 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 742b03ea3048466da3c9ea5ed9010b27 has no consensus state
W20260317 21:18:50.297478 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f41f18434a744b03b6ff53f3d7760e55 has no consensus state
I20260317 21:18:50.309408 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f280d2394ccd432ea164002a2b8ce9f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:50.311394 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f280d2394ccd432ea164002a2b8ce9f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.318864 6666 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 1 got 0
I20260317 21:18:50.321887 7462 ts_tablet_manager.cc:1434] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.106s user 0.005s sys 0.005s
I20260317 21:18:50.329533 7162 tablet_service.cc:1511] Processing CreateTablet for tablet f41f18434a744b03b6ff53f3d7760e55 (DEFAULT_TABLE table=default.Table_f671acaa7a4b41fcb385266ab7e21141 [id=6de336aaef3744f58b4fdfab525c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.330725 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41f18434a744b03b6ff53f3d7760e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.333469 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 1 got 0
I20260317 21:18:50.336189 7462 tablet_bootstrap.cc:492] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:50.341070 7163 tablet_service.cc:1511] Processing CreateTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (DEFAULT_TABLE table=default.Table_5c0d07d92b2d4bd3bfefa18fc6803f7a [id=2ed77cec958e4b008989a60fb67a2adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.342598 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019d90f6d70e46b087ea3c5ffd7cbd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.344538 7307 tablet_service.cc:1511] Processing CreateTablet for tablet f41f18434a744b03b6ff53f3d7760e55 (DEFAULT_TABLE table=default.Table_f671acaa7a4b41fcb385266ab7e21141 [id=6de336aaef3744f58b4fdfab525c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.345743 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41f18434a744b03b6ff53f3d7760e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.352213 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:50.355269 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:50.356132 7018 tablet_service.cc:1511] Processing CreateTablet for tablet f41f18434a744b03b6ff53f3d7760e55 (DEFAULT_TABLE table=default.Table_f671acaa7a4b41fcb385266ab7e21141 [id=6de336aaef3744f58b4fdfab525c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:50.357038 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 because the tablet was not found. No further retry: Not found: Tablet not found: 3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:50.358296 7018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f41f18434a744b03b6ff53f3d7760e55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.357535 7475 raft_consensus.cc:515] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.359521 6668 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 87a3f11d9d4d4037af60cf1a9b4f3377
I20260317 21:18:50.360059 7475 leader_election.cc:290] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:50.361632 6668 catalog_manager.cc:6741] ReplaceTablet: tablet 87a3f11d9d4d4037af60cf1a9b4f3377 deleted and replaced by tablet 33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:50.363194 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31725534b2a9401e9a79e8a467ed5210" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:50.364365 7038 raft_consensus.cc:3060] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:50.366566 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 with delete_type TABLET_DATA_DELETED (replaced by tablet 33f4e1fbb0d24813b0758cae9477f006) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:50.368670 7467 tablet_bootstrap.cc:492] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:50.369118 7467 ts_tablet_manager.cc:1403] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.235s user 0.030s sys 0.042s
W20260317 21:18:50.369477 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a3f11d9d4d4037af60cf1a9b4f3377 already in progress: creating tablet
I20260317 21:18:50.372560 7467 raft_consensus.cc:359] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.380543 7467 raft_consensus.cc:385] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.380959 7467 raft_consensus.cc:740] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.382934 7462 tablet_bootstrap.cc:654] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.386165 7467 consensus_queue.cc:260] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.399991 6669 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_7a6911ff81c44dbf87a617dd4ceeb1e6" } modify_external_catalogs: true
I20260317 21:18:50.399840 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a2ec133e1bdc4928889a9f16c9e502b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:50.400619 6669 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_7a6911ff81c44dbf87a617dd4ceeb1e6" } modify_external_catalogs: true
I20260317 21:18:50.401305 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_2fab17210bfa496ba34bb9580bee9d79"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:50.402387 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2ec133e1bdc4928889a9f16c9e502b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:50.403220 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2fab17210bfa496ba34bb9580bee9d79 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.404779 7467 ts_tablet_manager.cc:1434] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.035s user 0.010s sys 0.000s
I20260317 21:18:50.405848 7467 tablet_bootstrap.cc:492] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:50.405938 7163 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 with delete_type TABLET_DATA_DELETED (replaced by tablet 33f4e1fbb0d24813b0758cae9477f006) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:50.422425 7502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:50.407022 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1831269296644fc18ad3b11bdde06abd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:50.426868 7478 tablet_replica.cc:333] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
W20260317 21:18:50.427131 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1831269296644fc18ad3b11bdde06abd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.415649 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet fc0fe1bcaeb04f7eb157172f9c216d56
I20260317 21:18:50.430473 6672 catalog_manager.cc:6741] ReplaceTablet: tablet fc0fe1bcaeb04f7eb157172f9c216d56 deleted and replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3
I20260317 21:18:50.430473 7459 tablet_bootstrap.cc:654] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.434574 7467 tablet_bootstrap.cc:654] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.436489 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (DEFAULT_TABLE table=default.Table_5c0d07d92b2d4bd3bfefa18fc6803f7a [id=2ed77cec958e4b008989a60fb67a2adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.437718 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019d90f6d70e46b087ea3c5ffd7cbd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.437827 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 with delete_type TABLET_DATA_DELETED (replaced by tablet 33f4e1fbb0d24813b0758cae9477f006) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:50.438112 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:50.439756 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (DEFAULT_TABLE table=default.Table_5c0d07d92b2d4bd3bfefa18fc6803f7a [id=2ed77cec958e4b008989a60fb67a2adc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.441042 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 019d90f6d70e46b087ea3c5ffd7cbd28. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.447064 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:50.454753 7163 tablet_service.cc:1511] Processing CreateTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 (DEFAULT_TABLE table=default.Table_f3adbc8bf1994597bf7be6a6792ebf5a [id=16fb03a1fcfb4eff833f8ff6292c0b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.455272 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 with delete_type TABLET_DATA_DELETED (replaced by tablet 33f4e1fbb0d24813b0758cae9477f006) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:50.456023 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef4a015f5c14ee6afd045a87eaa6a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.456918 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:50.457059 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "31725534b2a9401e9a79e8a467ed5210" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:50.458238 7478 raft_consensus.cc:2243] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
W20260317 21:18:50.458741 7119 leader_election.cc:343] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:50.462718 7164 tablet_service.cc:1511] Processing CreateTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 (DEFAULT_TABLE table=default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.462909 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:50.463160 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 (DEFAULT_TABLE table=default.Table_f3adbc8bf1994597bf7be6a6792ebf5a [id=16fb03a1fcfb4eff833f8ff6292c0b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.463898 7483 tablet_replica.cc:333] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:50.464762 7483 raft_consensus.cc:2243] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:50.458763 7478 raft_consensus.cc:2272] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:50.465303 7483 raft_consensus.cc:2272] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:50.466609 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef4a015f5c14ee6afd045a87eaa6a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.467614 7478 ts_tablet_manager.cc:1916] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:50.467788 7483 ts_tablet_manager.cc:1916] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:50.464104 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6343bd5fc6a44ac3976cbbed8fab26c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.476962 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:50.477888 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 742b03ea3048466da3c9ea5ed9010b27 (DEFAULT_TABLE table=default.Table_488efb7758fb4a7a8bdbfae0f0864951 [id=bfdb4aafb98244168a6123384b829954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.479514 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742b03ea3048466da3c9ea5ed9010b27. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:50.488014 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a3f11d9d4d4037af60cf1a9b4f3377 already in progress: creating tablet
W20260317 21:18:50.488854 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 because the tablet was not found. No further retry: Not found: Tablet not found: 3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:50.490406 7462 tablet_bootstrap.cc:492] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.490895 7462 ts_tablet_manager.cc:1403] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.155s user 0.009s sys 0.020s
I20260317 21:18:50.495952 7307 tablet_service.cc:1511] Processing CreateTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 (DEFAULT_TABLE table=default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.497201 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6343bd5fc6a44ac3976cbbed8fab26c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.501652 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 742b03ea3048466da3c9ea5ed9010b27 (DEFAULT_TABLE table=default.Table_488efb7758fb4a7a8bdbfae0f0864951 [id=bfdb4aafb98244168a6123384b829954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.502562 7160 tablet_service.cc:1511] Processing CreateTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.503998 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742b03ea3048466da3c9ea5ed9010b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.504245 7160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42fa238590b47d5bc0bd43dd3a313f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.507503 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_488efb7758fb4a7a8bdbfae0f0864951" } new_table_name: "default.Table_eec7ee55d8bf4e9baa04680fd31ac46c" modify_external_catalogs: true
I20260317 21:18:50.508108 7462 raft_consensus.cc:359] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.508890 7462 raft_consensus.cc:385] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.510463 7305 tablet_service.cc:1511] Processing CreateTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.512292 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42fa238590b47d5bc0bd43dd3a313f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.513185 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f671acaa7a4b41fcb385266ab7e21141" } new_table_name: "default.Table_8e6a77924a3748cbbb655ae7c443cfb4" modify_external_catalogs: true
I20260317 21:18:50.509200 7462 raft_consensus.cc:740] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.527215 7462 consensus_queue.cc:260] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.528564 7467 tablet_bootstrap.cc:492] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:50.529069 7467 ts_tablet_manager.cc:1403] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.123s user 0.009s sys 0.005s
I20260317 21:18:50.530478 6669 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 98fdf22ad21f4b528532745b2def5cbc
I20260317 21:18:50.532158 7462 ts_tablet_manager.cc:1434] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.040s user 0.008s sys 0.002s
I20260317 21:18:50.543418 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 (DEFAULT_TABLE table=default.Table_f3adbc8bf1994597bf7be6a6792ebf5a [id=16fb03a1fcfb4eff833f8ff6292c0b21]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.546639 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 (DEFAULT_TABLE table=default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.554579 7462 tablet_bootstrap.cc:492] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:50.556209 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:50.562800 7011 tablet_service.cc:1511] Processing CreateTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.567874 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 742b03ea3048466da3c9ea5ed9010b27 (DEFAULT_TABLE table=default.Table_488efb7758fb4a7a8bdbfae0f0864951 [id=bfdb4aafb98244168a6123384b829954]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.572158 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6343bd5fc6a44ac3976cbbed8fab26c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.573985 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 742b03ea3048466da3c9ea5ed9010b27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.575902 7462 tablet_bootstrap.cc:654] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.584992 7011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b42fa238590b47d5bc0bd43dd3a313f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.595382 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3ef4a015f5c14ee6afd045a87eaa6a89. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.595247 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_133ff1a27c134c24b1d11b25918e902d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:50.597747 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_133ff1a27c134c24b1d11b25918e902d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.597963 7462 tablet_bootstrap.cc:492] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.598584 7462 ts_tablet_manager.cc:1403] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.045s user 0.012s sys 0.000s
I20260317 21:18:50.602447 7462 raft_consensus.cc:359] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.603466 7462 raft_consensus.cc:385] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.603822 7462 raft_consensus.cc:740] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.604928 7462 consensus_queue.cc:260] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.617408 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 with delete_type TABLET_DATA_DELETED (replaced by tablet 33f4e1fbb0d24813b0758cae9477f006) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:50.620597 7038 raft_consensus.cc:2468] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:50.621421 7462 ts_tablet_manager.cc:1434] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.022s user 0.007s sys 0.003s
I20260317 21:18:50.621856 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_9f87958031d24bfbb6e275b81361fc4e" } modify_external_catalogs: true
I20260317 21:18:50.625615 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_9f87958031d24bfbb6e275b81361fc4e" } modify_external_catalogs: true
I20260317 21:18:50.625212 7483 ts_tablet_manager.cc:1929] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:50.637624 7483 log.cc:1199] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/fc0fe1bcaeb04f7eb157172f9c216d56
I20260317 21:18:50.639810 7462 tablet_bootstrap.cc:492] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:50.645690 7483 ts_tablet_manager.cc:1950] T fc0fe1bcaeb04f7eb157172f9c216d56 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:50.653522 7483 tablet_replica.cc:333] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:50.654354 7483 raft_consensus.cc:2243] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:50.655432 7483 raft_consensus.cc:2272] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:50.660220 7462 tablet_bootstrap.cc:654] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.663275 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet fc0fe1bcaeb04f7eb157172f9c216d56 (table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]) successfully deleted
W20260317 21:18:50.664508 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet fc0fe1bcaeb04f7eb157172f9c216d56 because the tablet was not found. No further retry: Not found: Tablet not found: fc0fe1bcaeb04f7eb157172f9c216d56
I20260317 21:18:50.666482 7483 ts_tablet_manager.cc:1916] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:50.667398 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet 31725534b2a9401e9a79e8a467ed5210 from {username='slave'} at 127.0.0.1:39038
W20260317 21:18:50.670020 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f41f18434a744b03b6ff53f3d7760e55 consensus state has no leader
W20260317 21:18:50.673209 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 742b03ea3048466da3c9ea5ed9010b27 consensus state has no leader
I20260317 21:18:50.681967 6666 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 1 got 0
I20260317 21:18:50.532050 7467 raft_consensus.cc:359] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.646109 7120 leader_election.cc:304] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:50.685201 7475 raft_consensus.cc:2804] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:50.685390 7467 raft_consensus.cc:385] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.685848 7467 raft_consensus.cc:740] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.686497 6670 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 1 got 0
I20260317 21:18:50.686587 7467 consensus_queue.cc:260] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.700902 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 3360925bb5e2415db39a551800ca28d0 (DEFAULT_TABLE table=default.Table_a7b4fa95d87c4a10895ca301321c5738 [id=7680260305df414a9e15cd0f1f47a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.702956 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3360925bb5e2415db39a551800ca28d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.724571 7483 ts_tablet_manager.cc:1929] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:50.725116 7483 log.cc:1199] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/87a3f11d9d4d4037af60cf1a9b4f3377
I20260317 21:18:50.726192 7462 tablet_bootstrap.cc:492] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.726642 7462 ts_tablet_manager.cc:1403] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.087s user 0.025s sys 0.008s
I20260317 21:18:50.730047 7462 raft_consensus.cc:359] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.731199 7462 raft_consensus.cc:385] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.731688 7462 raft_consensus.cc:740] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.732764 7462 consensus_queue.cc:260] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.735013 7462 ts_tablet_manager.cc:1434] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.008s user 0.004s sys 0.002s
I20260317 21:18:50.736570 7017 tablet_service.cc:1511] Processing CreateTablet for tablet 3360925bb5e2415db39a551800ca28d0 (DEFAULT_TABLE table=default.Table_a7b4fa95d87c4a10895ca301321c5738 [id=7680260305df414a9e15cd0f1f47a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.738040 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3360925bb5e2415db39a551800ca28d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.725945 7483 ts_tablet_manager.cc:1950] T 87a3f11d9d4d4037af60cf1a9b4f3377 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:50.737623 7010 tablet_service.cc:1511] Processing CreateTablet for tablet bc93b8e8da49471493369deef1a502f3 (DEFAULT_TABLE table=default.Table_9f87958031d24bfbb6e275b81361fc4e [id=be78d528072d4de29b4c27d71d6f7034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.739081 7462 tablet_bootstrap.cc:492] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:50.741061 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 (DEFAULT_TABLE table=default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.747514 7462 tablet_bootstrap.cc:654] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.760524 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 (DEFAULT_TABLE table=default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.761896 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8deb47bf1eee48a09d8fd6543e409ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.763469 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 (DEFAULT_TABLE table=default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.766870 7157 tablet_service.cc:1511] Processing CreateTablet for tablet bc93b8e8da49471493369deef1a502f3 (DEFAULT_TABLE table=default.Table_9f87958031d24bfbb6e275b81361fc4e [id=be78d528072d4de29b4c27d71d6f7034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.774039 7462 tablet_bootstrap.cc:492] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.773725 7008 tablet_service.cc:1511] Processing CreateTablet for tablet f0d55550bf914019b83e83fa4ffcb64b (DEFAULT_TABLE table=default.Table_5bd5abc0a4a64c408b1bae33441413d4 [id=577b3aea931d4e518d588b81917b49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.768613 7157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc93b8e8da49471493369deef1a502f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.775331 7462 ts_tablet_manager.cc:1403] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.036s user 0.010s sys 0.007s
I20260317 21:18:50.776068 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc93b8e8da49471493369deef1a502f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.779012 7462 raft_consensus.cc:359] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.780829 7462 raft_consensus.cc:385] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.781234 7462 raft_consensus.cc:740] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.781885 7462 consensus_queue.cc:260] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.769812 7477 ts_tablet_manager.cc:1929] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:50.789106 7477 log.cc:1199] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/00e7af4aa99341189d3001c9188689d7
I20260317 21:18:50.789885 7477 ts_tablet_manager.cc:1950] T 00e7af4aa99341189d3001c9188689d7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:50.790863 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 3360925bb5e2415db39a551800ca28d0 (DEFAULT_TABLE table=default.Table_a7b4fa95d87c4a10895ca301321c5738 [id=7680260305df414a9e15cd0f1f47a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.792618 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3360925bb5e2415db39a551800ca28d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.795059 7467 ts_tablet_manager.cc:1434] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.266s user 0.009s sys 0.010s
I20260317 21:18:50.796386 7467 tablet_bootstrap.cc:492] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:50.797145 7462 ts_tablet_manager.cc:1434] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.021s user 0.006s sys 0.000s
I20260317 21:18:50.799172 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 27977b00ae1a4e7ca2eddaa693d11a65 (DEFAULT_TABLE table=default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.807786 7462 tablet_bootstrap.cc:492] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:50.808784 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8deb47bf1eee48a09d8fd6543e409ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.773696 7475 raft_consensus.cc:697] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:50.814041 7307 tablet_service.cc:1511] Processing CreateTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 (DEFAULT_TABLE table=default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.815289 7462 tablet_bootstrap.cc:654] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.815618 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8deb47bf1eee48a09d8fd6543e409ec3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.827270 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d55550bf914019b83e83fa4ffcb64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.831621 7305 tablet_service.cc:1511] Processing CreateTablet for tablet bc93b8e8da49471493369deef1a502f3 (DEFAULT_TABLE table=default.Table_9f87958031d24bfbb6e275b81361fc4e [id=be78d528072d4de29b4c27d71d6f7034]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.833235 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc93b8e8da49471493369deef1a502f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.833597 7462 tablet_bootstrap.cc:492] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.834026 7462 ts_tablet_manager.cc:1403] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.014s
I20260317 21:18:50.835516 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27977b00ae1a4e7ca2eddaa693d11a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.837105 7462 raft_consensus.cc:359] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.837828 7462 raft_consensus.cc:385] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.838124 7462 raft_consensus.cc:740] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.838773 7462 consensus_queue.cc:260] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.841637 7459 tablet_bootstrap.cc:492] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:50.842186 7459 ts_tablet_manager.cc:1403] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.579s user 0.007s sys 0.014s
I20260317 21:18:50.845464 7459 raft_consensus.cc:359] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.848929 7462 ts_tablet_manager.cc:1434] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.015s user 0.008s sys 0.000s
I20260317 21:18:50.855232 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 (DEFAULT_TABLE table=default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.856829 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f4e1fbb0d24813b0758cae9477f006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.857163 7462 tablet_bootstrap.cc:492] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:18:50.860020 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 because the tablet was not found. No further retry: Not found: Tablet not found: 3ef4a015f5c14ee6afd045a87eaa6a89
W20260317 21:18:50.860874 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 87a3f11d9d4d4037af60cf1a9b4f3377 already in progress: creating tablet
W20260317 21:18:50.862610 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet fc0fe1bcaeb04f7eb157172f9c216d56 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc0fe1bcaeb04f7eb157172f9c216d56 already in progress: creating tablet
I20260317 21:18:50.878448 7498 raft_consensus.cc:493] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:50.879228 7498 raft_consensus.cc:515] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.885653 7462 tablet_bootstrap.cc:654] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:50.886097 7308 tablet_service.cc:1511] Processing CreateTablet for tablet f0d55550bf914019b83e83fa4ffcb64b (DEFAULT_TABLE table=default.Table_5bd5abc0a4a64c408b1bae33441413d4 [id=577b3aea931d4e518d588b81917b49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.890736 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 87a3f11d9d4d4037af60cf1a9b4f3377 (table default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]) successfully deleted
I20260317 21:18:50.892983 6670 master_service.cc:954] ReplaceTablet: received request to replace tablet 87a3f11d9d4d4037af60cf1a9b4f3377 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:50.893903 7155 tablet_service.cc:1511] Processing CreateTablet for tablet f0d55550bf914019b83e83fa4ffcb64b (DEFAULT_TABLE table=default.Table_5bd5abc0a4a64c408b1bae33441413d4 [id=577b3aea931d4e518d588b81917b49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.894517 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_9f4465b6fdf24369a96e3bf37cfd19d7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:50.856851 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f4e1fbb0d24813b0758cae9477f006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.895943 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d55550bf914019b83e83fa4ffcb64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.888756 7498 leader_election.cc:290] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:50.897131 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_2f865cde34f646c6b0058233b3bbc3a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:50.881114 7156 tablet_service.cc:1511] Processing CreateTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 (DEFAULT_TABLE table=default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:50.899065 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2f865cde34f646c6b0058233b3bbc3a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.898741 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f0d55550bf914019b83e83fa4ffcb64b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.899991 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_5fce106f6b6a4e00b9af07d5e5fc930f" } modify_external_catalogs: true
I20260317 21:18:50.900522 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_5fce106f6b6a4e00b9af07d5e5fc930f" } modify_external_catalogs: true
W20260317 21:18:50.903359 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f4465b6fdf24369a96e3bf37cfd19d7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.919811 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b1d7ee6663fe432a8940df97de65d99b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:50.921757 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1d7ee6663fe432a8940df97de65d99b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:50.920746 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 27977b00ae1a4e7ca2eddaa693d11a65 (DEFAULT_TABLE table=default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.935410 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 27977b00ae1a4e7ca2eddaa693d11a65 (DEFAULT_TABLE table=default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:50.937439 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27977b00ae1a4e7ca2eddaa693d11a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.945396 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 33f4e1fbb0d24813b0758cae9477f006. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.959219 7462 tablet_bootstrap.cc:492] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:50.959810 7462 ts_tablet_manager.cc:1403] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.103s user 0.012s sys 0.001s
I20260317 21:18:50.962350 7462 raft_consensus.cc:359] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.963127 7462 raft_consensus.cc:385] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:50.963605 7462 raft_consensus.cc:740] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:50.972869 7478 ts_tablet_manager.cc:1929] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:50.973502 7478 log.cc:1199] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/87a3f11d9d4d4037af60cf1a9b4f3377
I20260317 21:18:50.974387 7478 ts_tablet_manager.cc:1950] T 87a3f11d9d4d4037af60cf1a9b4f3377 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:50.984897 7475 consensus_queue.cc:237] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:50.994865 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_a7b4fa95d87c4a10895ca301321c5738" } new_table_name: "default.Table_8f5494345b90406b91c999701c781c58" modify_external_catalogs: true
I20260317 21:18:50.996395 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 27977b00ae1a4e7ca2eddaa693d11a65. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:50.997480 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_a2ec133e1bdc4928889a9f16c9e502b8" } new_table_name: "default.Table_ffede0d29ebf494f8eec15158eeea6f6" modify_external_catalogs: true
I20260317 21:18:50.964257 7462 consensus_queue.cc:260] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.030129 7462 ts_tablet_manager.cc:1434] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20260317 21:18:51.031075 7462 tablet_bootstrap.cc:492] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:51.040347 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 87a3f11d9d4d4037af60cf1a9b4f3377 (table default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]) successfully deleted
I20260317 21:18:51.018766 7459 raft_consensus.cc:385] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.054425 7462 tablet_bootstrap.cc:654] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.041129 7459 raft_consensus.cc:740] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.057390 7459 consensus_queue.cc:260] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.060801 7478 tablet_replica.cc:333] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:51.062397 7478 raft_consensus.cc:2243] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:51.064546 7498 raft_consensus.cc:493] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:51.065117 7498 raft_consensus.cc:515] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.066363 7478 raft_consensus.cc:2272] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.067190 7498 leader_election.cc:290] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:51.075994 7478 ts_tablet_manager.cc:1916] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.106432 7462 tablet_bootstrap.cc:492] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:51.107342 7462 ts_tablet_manager.cc:1403] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.076s user 0.012s sys 0.000s
I20260317 21:18:51.109704 7462 raft_consensus.cc:359] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.110379 7462 raft_consensus.cc:385] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.110620 7462 raft_consensus.cc:740] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.111186 7459 ts_tablet_manager.cc:1434] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.268s user 0.008s sys 0.004s
I20260317 21:18:51.111385 7462 consensus_queue.cc:260] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.113957 7459 tablet_bootstrap.cc:492] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:51.129534 7462 ts_tablet_manager.cc:1434] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.022s user 0.008s sys 0.000s
I20260317 21:18:51.130539 7462 tablet_bootstrap.cc:492] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:51.145776 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 87a3f11d9d4d4037af60cf1a9b4f3377 with delete_type TABLET_DATA_DELETED (replaced by tablet 33f4e1fbb0d24813b0758cae9477f006) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:51.146696 7477 tablet_replica.cc:333] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:51.147524 7477 raft_consensus.cc:2243] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:51.148030 7477 raft_consensus.cc:2272] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:18:51.149631 6662 consensus_peers.cc:597] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 -> Peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Couldn't send request to peer 47e1c983d0224428a9a7b147a0e5e4ef. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20260317 21:18:51.061612 7467 tablet_bootstrap.cc:654] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.180533 7467 tablet_bootstrap.cc:492] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:51.180351 7477 ts_tablet_manager.cc:1916] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.181030 7467 ts_tablet_manager.cc:1403] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.385s user 0.010s sys 0.004s
I20260317 21:18:51.183529 7467 raft_consensus.cc:359] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.184273 7467 raft_consensus.cc:385] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.184600 7467 raft_consensus.cc:740] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.186679 7462 tablet_bootstrap.cc:654] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.190212 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet bc93b8e8da49471493369deef1a502f3
I20260317 21:18:51.195520 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet bc93b8e8da49471493369deef1a502f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:51.195688 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet bc93b8e8da49471493369deef1a502f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:51.197535 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet bc93b8e8da49471493369deef1a502f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc93b8e8da49471493369deef1a502f3 already in progress: creating tablet
I20260317 21:18:51.200217 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 31725534b2a9401e9a79e8a467ed5210
I20260317 21:18:51.200582 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:51.202443 6672 catalog_manager.cc:6741] ReplaceTablet: tablet 31725534b2a9401e9a79e8a467ed5210 deleted and replaced by tablet 06877bd7155842b9a5ae4a23ea68ada0
I20260317 21:18:51.202245 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6343bd5fc6a44ac3976cbbed8fab26c7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:51.203860 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 31725534b2a9401e9a79e8a467ed5210 with delete_type TABLET_DATA_DELETED (replaced by tablet 06877bd7155842b9a5ae4a23ea68ada0) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.206168 6666 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 2 got 0
I20260317 21:18:51.208348 7467 consensus_queue.cc:260] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:51.213722 7261 leader_election.cc:343] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.214133 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 31725534b2a9401e9a79e8a467ed5210 with delete_type TABLET_DATA_DELETED (replaced by tablet 06877bd7155842b9a5ae4a23ea68ada0) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:51.214591 7261 leader_election.cc:343] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.211000 7498 raft_consensus.cc:493] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:51.215451 7498 raft_consensus.cc:515] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.216087 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 31725534b2a9401e9a79e8a467ed5210 with delete_type TABLET_DATA_DELETED (replaced by tablet 06877bd7155842b9a5ae4a23ea68ada0) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.216531 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet bc93b8e8da49471493369deef1a502f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.217602 7498 leader_election.cc:290] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:51.218602 7483 tablet_replica.cc:333] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:51.219676 7483 raft_consensus.cc:2243] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:51.220203 7483 raft_consensus.cc:2272] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.223829 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef4a015f5c14ee6afd045a87eaa6a89" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:51.227152 7467 ts_tablet_manager.cc:1434] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.046s user 0.004s sys 0.002s
I20260317 21:18:51.230409 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet 3360925bb5e2415db39a551800ca28d0 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:51.231534 7467 tablet_bootstrap.cc:492] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:51.233443 7459 tablet_bootstrap.cc:654] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:51.251528 7261 leader_election.cc:343] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.252710 7483 ts_tablet_manager.cc:1916] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.262916 7478 ts_tablet_manager.cc:1929] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:51.263353 7478 log.cc:1199] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/fc0fe1bcaeb04f7eb157172f9c216d56
I20260317 21:18:51.264038 7478 ts_tablet_manager.cc:1950] T fc0fe1bcaeb04f7eb157172f9c216d56 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:51.267190 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet fc0fe1bcaeb04f7eb157172f9c216d56 (table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]) successfully deleted
I20260317 21:18:51.277001 7478 tablet_replica.cc:333] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
W20260317 21:18:51.277630 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet f41f18434a744b03b6ff53f3d7760e55 consensus state has no leader
I20260317 21:18:51.278100 7478 raft_consensus.cc:2243] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:51.279275 7478 pending_rounds.cc:70] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:18:51.280051 7462 tablet_bootstrap.cc:492] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:51.280478 7462 ts_tablet_manager.cc:1403] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.150s user 0.019s sys 0.004s
I20260317 21:18:51.283402 7462 raft_consensus.cc:359] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.284116 7462 raft_consensus.cc:385] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.280066 7478 pending_rounds.cc:77] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412427807993856 op_type: NO_OP noop_request { }
I20260317 21:18:51.285080 7478 raft_consensus.cc:2889] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:51.285564 7478 raft_consensus.cc:2272] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.288463 7478 ts_tablet_manager.cc:1916] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.289422 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_93723f1459114d9ba82a21cee9158b3d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:51.292201 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93723f1459114d9ba82a21cee9158b3d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.301152 7467 tablet_bootstrap.cc:654] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.308754 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6343bd5fc6a44ac3976cbbed8fab26c7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:51.310717 7264 leader_election.cc:343] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:51.311246 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be2c469c792047d88fcf2ff9d53b7832 has no consensus state
I20260317 21:18:51.311177 7264 leader_election.cc:304] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:51.310745 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef4a015f5c14ee6afd045a87eaa6a89" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:51.284407 7462 raft_consensus.cc:740] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.312839 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:51.313169 7462 consensus_queue.cc:260] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:51.314673 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3360925bb5e2415db39a551800ca28d0 consensus state has no leader
I20260317 21:18:51.314956 7498 raft_consensus.cc:2749] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:51.316844 7264 leader_election.cc:343] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.317397 7264 leader_election.cc:304] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:51.318418 7498 raft_consensus.cc:2749] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:51.319854 7264 leader_election.cc:343] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.320376 7264 leader_election.cc:304] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:51.321810 7498 raft_consensus.cc:2749] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:51.322928 7483 ts_tablet_manager.cc:1929] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:51.323310 7483 log.cc:1199] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/31725534b2a9401e9a79e8a467ed5210
I20260317 21:18:51.324074 7483 ts_tablet_manager.cc:1950] T 31725534b2a9401e9a79e8a467ed5210 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:51.326900 7467 tablet_bootstrap.cc:492] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:51.327343 7467 ts_tablet_manager.cc:1403] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.000s
I20260317 21:18:51.329752 7467 raft_consensus.cc:359] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.330418 7467 raft_consensus.cc:385] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.333292 7462 ts_tablet_manager.cc:1434] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.052s user 0.006s sys 0.000s
I20260317 21:18:51.336843 7478 ts_tablet_manager.cc:1929] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:51.337296 7478 log.cc:1199] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/31725534b2a9401e9a79e8a467ed5210
I20260317 21:18:51.338114 7478 ts_tablet_manager.cc:1950] T 31725534b2a9401e9a79e8a467ed5210 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:51.340420 7459 tablet_bootstrap.cc:492] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:51.340919 7459 ts_tablet_manager.cc:1403] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.227s user 0.009s sys 0.004s
I20260317 21:18:51.340895 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]) successfully deleted
I20260317 21:18:51.344594 7477 ts_tablet_manager.cc:1929] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:51.330679 7467 raft_consensus.cc:740] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.351171 7467 consensus_queue.cc:260] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.353564 7477 log.cc:1199] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/87a3f11d9d4d4037af60cf1a9b4f3377
I20260317 21:18:51.354377 7477 ts_tablet_manager.cc:1950] T 87a3f11d9d4d4037af60cf1a9b4f3377 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:51.358781 7467 ts_tablet_manager.cc:1434] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.031s user 0.007s sys 0.001s
I20260317 21:18:51.359447 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 8deb47bf1eee48a09d8fd6543e409ec3
I20260317 21:18:51.359727 7467 tablet_bootstrap.cc:492] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:51.362839 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.365163 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 87a3f11d9d4d4037af60cf1a9b4f3377 (table default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]) successfully deleted
I20260317 21:18:51.367370 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.368106 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.368294 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:51.368906 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet bc93b8e8da49471493369deef1a502f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc93b8e8da49471493369deef1a502f3 already in progress: creating tablet
I20260317 21:18:51.368772 7154 tablet_service.cc:1511] Processing CreateTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a (DEFAULT_TABLE table=default.Table_1831269296644fc18ad3b11bdde06abd [id=8a0a0d875f5e4f5fbcbb74938698fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.369918 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b196f1e1572f4a62b45df7ffc581cd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.372478 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.375216 7008 tablet_service.cc:1511] Processing CreateTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a (DEFAULT_TABLE table=default.Table_1831269296644fc18ad3b11bdde06abd [id=8a0a0d875f5e4f5fbcbb74938698fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.376330 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b196f1e1572f4a62b45df7ffc581cd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.377331 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.377936 7007 tablet_service.cc:1511] Processing CreateTablet for tablet be2c469c792047d88fcf2ff9d53b7832 (DEFAULT_TABLE table=default.Table_ffede0d29ebf494f8eec15158eeea6f6 [id=e9385f5ddf6e430a83595a74932a704e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.379496 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2c469c792047d88fcf2ff9d53b7832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.381510 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 31725534b2a9401e9a79e8a467ed5210 with delete_type TABLET_DATA_DELETED (replaced by tablet 06877bd7155842b9a5ae4a23ea68ada0) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.383620 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 291e2bee709243f09b338d99edbe0cb1 (DEFAULT_TABLE table=default.Table_2fab17210bfa496ba34bb9580bee9d79 [id=040d2e5e61ef41e4ac8953a3d63a226c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.385478 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 291e2bee709243f09b338d99edbe0cb1 (DEFAULT_TABLE table=default.Table_2fab17210bfa496ba34bb9580bee9d79 [id=040d2e5e61ef41e4ac8953a3d63a226c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.386775 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291e2bee709243f09b338d99edbe0cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.386831 7462 tablet_bootstrap.cc:492] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:51.387063 7308 tablet_service.cc:1511] Processing CreateTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a (DEFAULT_TABLE table=default.Table_1831269296644fc18ad3b11bdde06abd [id=8a0a0d875f5e4f5fbcbb74938698fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.387957 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291e2bee709243f09b338d99edbe0cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.389936 7304 tablet_service.cc:1511] Processing CreateTablet for tablet be2c469c792047d88fcf2ff9d53b7832 (DEFAULT_TABLE table=default.Table_ffede0d29ebf494f8eec15158eeea6f6 [id=e9385f5ddf6e430a83595a74932a704e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.393236 7462 tablet_bootstrap.cc:654] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.395046 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b196f1e1572f4a62b45df7ffc581cd7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.396752 7155 tablet_service.cc:1511] Processing CreateTablet for tablet be2c469c792047d88fcf2ff9d53b7832 (DEFAULT_TABLE table=default.Table_ffede0d29ebf494f8eec15158eeea6f6 [id=e9385f5ddf6e430a83595a74932a704e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.397576 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2c469c792047d88fcf2ff9d53b7832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.397898 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be2c469c792047d88fcf2ff9d53b7832. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.399250 7483 tablet_replica.cc:333] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:51.400067 7483 raft_consensus.cc:2243] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:51.402285 7467 tablet_bootstrap.cc:654] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:51.402884 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 8deb47bf1eee48a09d8fd6543e409ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8deb47bf1eee48a09d8fd6543e409ec3 already in progress: creating tablet
W20260317 21:18:51.403734 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet fc0fe1bcaeb04f7eb157172f9c216d56 because the tablet was not found. No further retry: Not found: Tablet not found: fc0fe1bcaeb04f7eb157172f9c216d56
W20260317 21:18:51.404549 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 31725534b2a9401e9a79e8a467ed5210 because the tablet was not found. No further retry: Not found: Tablet not found: 31725534b2a9401e9a79e8a467ed5210
I20260317 21:18:51.406447 6668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_844e7186ad624a5da968ffef9aac1610" } modify_external_catalogs: true
I20260317 21:18:51.407178 6668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_844e7186ad624a5da968ffef9aac1610" } modify_external_catalogs: true
I20260317 21:18:51.411010 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 3360925bb5e2415db39a551800ca28d0
I20260317 21:18:51.413249 6666 catalog_manager.cc:6741] ReplaceTablet: tablet 3360925bb5e2415db39a551800ca28d0 deleted and replaced by tablet 60f3d32e1f7649fa92861f51134cf000
I20260317 21:18:51.417968 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_562f2136d87742b4839f28ff31c05f40 [id=405c1ee68fd5448e87ca58a307644859]) successfully deleted
W20260317 21:18:51.418789 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 8deb47bf1eee48a09d8fd6543e409ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8deb47bf1eee48a09d8fd6543e409ec3 already in progress: creating tablet
I20260317 21:18:51.419725 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet 3360925bb5e2415db39a551800ca28d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 60f3d32e1f7649fa92861f51134cf000) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:51.426476 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 291e2bee709243f09b338d99edbe0cb1 (DEFAULT_TABLE table=default.Table_2fab17210bfa496ba34bb9580bee9d79 [id=040d2e5e61ef41e4ac8953a3d63a226c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.427826 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 291e2bee709243f09b338d99edbe0cb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.427806 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 3360925bb5e2415db39a551800ca28d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 60f3d32e1f7649fa92861f51134cf000) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.434376 7462 tablet_bootstrap.cc:492] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:51.436524 7467 tablet_bootstrap.cc:492] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:51.436996 7467 ts_tablet_manager.cc:1403] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.001s
I20260317 21:18:51.400556 7483 raft_consensus.cc:2272] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.437412 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet 3360925bb5e2415db39a551800ca28d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 60f3d32e1f7649fa92861f51134cf000) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.407887 6669 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954]). Expected version 3 got 0
I20260317 21:18:51.437119 7462 ts_tablet_manager.cc:1403] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.050s user 0.011s sys 0.000s
I20260317 21:18:51.441643 7483 ts_tablet_manager.cc:1916] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.442299 7467 raft_consensus.cc:359] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.443010 7467 raft_consensus.cc:385] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.443282 7467 raft_consensus.cc:740] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.444190 7467 consensus_queue.cc:260] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.446096 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f025e5d87c5f4a119d195af00e00dd38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:51.448972 6669 master_service.cc:954] ReplaceTablet: received request to replace tablet 6343bd5fc6a44ac3976cbbed8fab26c7 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:51.452555 7467 ts_tablet_manager.cc:1434] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260317 21:18:51.455797 7467 tablet_bootstrap.cc:492] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:51.352607 7459 raft_consensus.cc:359] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.456840 7459 raft_consensus.cc:385] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.457190 7459 raft_consensus.cc:740] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.458036 7459 consensus_queue.cc:260] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.463256 7462 raft_consensus.cc:359] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:51.453459 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f025e5d87c5f4a119d195af00e00dd38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:51.454159 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 3360925bb5e2415db39a551800ca28d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3360925bb5e2415db39a551800ca28d0 already in progress: creating tablet
I20260317 21:18:51.455611 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_49f59ddc9db844c5bcfb828a4735238b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:51.464339 7462 raft_consensus.cc:385] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.464787 7462 raft_consensus.cc:740] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:18:51.465881 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_49f59ddc9db844c5bcfb828a4735238b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.465480 7462 consensus_queue.cc:260] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.456708 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954]). Expected version 3 got 0
I20260317 21:18:51.495093 7462 ts_tablet_manager.cc:1434] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.056s user 0.007s sys 0.000s
I20260317 21:18:51.507772 7462 tablet_bootstrap.cc:492] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:18:51.509610 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet bc93b8e8da49471493369deef1a502f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bc93b8e8da49471493369deef1a502f3 already in progress: creating tablet
I20260317 21:18:51.511332 7477 tablet_replica.cc:333] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:51.515862 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet fc0fe1bcaeb04f7eb157172f9c216d56 with delete_type TABLET_DATA_DELETED (replaced by tablet 8deb47bf1eee48a09d8fd6543e409ec3) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:51.517202 6668 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:51.519083 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_74e46c4c417f4c7f95fbf9f9df94be0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:51.520771 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:51.521147 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_74e46c4c417f4c7f95fbf9f9df94be0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.522480 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:51.524827 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 33f4e1fbb0d24813b0758cae9477f006 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33f4e1fbb0d24813b0758cae9477f006 already in progress: creating tablet
I20260317 21:18:51.526917 7475 raft_consensus.cc:493] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:51.527709 7475 raft_consensus.cc:515] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.534509 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:51.535575 7329 raft_consensus.cc:2468] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:51.538345 7475 leader_election.cc:290] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:51.538870 7475 raft_consensus.cc:493] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:51.539552 7475 raft_consensus.cc:515] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.545406 7119 leader_election.cc:304] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:51.545749 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:51.547142 7120 leader_election.cc:343] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.550316 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.549995 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef4a015f5c14ee6afd045a87eaa6a89" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:51.549988 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3ef4a015f5c14ee6afd045a87eaa6a89" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:51.551878 7119 leader_election.cc:343] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:51.552465 7538 raft_consensus.cc:2804] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:51.552457 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_562f2136d87742b4839f28ff31c05f40" } new_table_name: "default.Table_20cf24578433433a96ad72f0593ca863" modify_external_catalogs: true
I20260317 21:18:51.552634 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_8f5494345b90406b91c999701c781c58" } new_table_name: "default.Table_7a69602e996b4ca6aa936f73b9b24c34" modify_external_catalogs: true
I20260317 21:18:51.552911 7538 raft_consensus.cc:493] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:51.553078 7467 tablet_bootstrap.cc:654] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.553261 7538 raft_consensus.cc:3060] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:51.552500 7475 leader_election.cc:290] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:51.557241 7120 leader_election.cc:343] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.560324 7120 leader_election.cc:304] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:51.561219 7459 ts_tablet_manager.cc:1434] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.220s user 0.006s sys 0.003s
I20260317 21:18:51.565776 7475 raft_consensus.cc:2749] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:51.572290 7459 tablet_bootstrap.cc:492] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:51.575964 7477 raft_consensus.cc:2243] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:51.582368 7538 raft_consensus.cc:515] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.586024 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
W20260317 21:18:51.587572 7120 leader_election.cc:343] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.590168 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_93723f1459114d9ba82a21cee9158b3d" } modify_external_catalogs: true
I20260317 21:18:51.590926 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_93723f1459114d9ba82a21cee9158b3d" } modify_external_catalogs: true
I20260317 21:18:51.592767 7477 raft_consensus.cc:2272] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.593595 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_6dc2ff9e049443948de3753522e042f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:51.595844 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6dc2ff9e049443948de3753522e042f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.596122 7477 ts_tablet_manager.cc:1916] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.597066 7538 leader_election.cc:290] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:51.626890 6669 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 6343bd5fc6a44ac3976cbbed8fab26c7
I20260317 21:18:51.631011 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7377c42c199b408daa7f7e730ec5db06) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:51.631771 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7377c42c199b408daa7f7e730ec5db06) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.561893 7483 ts_tablet_manager.cc:1929] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:51.598774 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:51.633682 7329 raft_consensus.cc:3060] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:51.633525 7483 log.cc:1199] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/3ef4a015f5c14ee6afd045a87eaa6a89
W20260317 21:18:51.637897 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6343bd5fc6a44ac3976cbbed8fab26c7 already in progress: creating tablet
I20260317 21:18:51.639355 7483 ts_tablet_manager.cc:1950] T 3ef4a015f5c14ee6afd045a87eaa6a89 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:51.643702 6669 catalog_manager.cc:6741] ReplaceTablet: tablet 6343bd5fc6a44ac3976cbbed8fab26c7 deleted and replaced by tablet 7377c42c199b408daa7f7e730ec5db06
I20260317 21:18:51.645454 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7377c42c199b408daa7f7e730ec5db06) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:51.645692 7329 raft_consensus.cc:2468] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:51.646929 7119 leader_election.cc:304] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:51.647802 7538 raft_consensus.cc:2804] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:51.610044 7462 tablet_bootstrap.cc:654] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.648253 7538 raft_consensus.cc:697] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:51.649441 7538 consensus_queue.cc:237] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.655578 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 3ef4a015f5c14ee6afd045a87eaa6a89 (table default.Table_f3adbc8bf1994597bf7be6a6792ebf5a [id=16fb03a1fcfb4eff833f8ff6292c0b21]) successfully deleted
I20260317 21:18:51.659207 7467 tablet_bootstrap.cc:492] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:51.659675 7467 ts_tablet_manager.cc:1403] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.204s user 0.013s sys 0.000s
I20260317 21:18:51.660061 7483 tablet_replica.cc:333] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:51.661641 7483 raft_consensus.cc:2243] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260317 21:18:51.667237 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 06877bd7155842b9a5ae4a23ea68ada0 has no consensus state
W20260317 21:18:51.668869 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 60f3d32e1f7649fa92861f51134cf000 has no consensus state
I20260317 21:18:51.671231 7467 raft_consensus.cc:359] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.672254 7467 raft_consensus.cc:385] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.676630 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet fc0fe1bcaeb04f7eb157172f9c216d56 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:51.688886 7483 raft_consensus.cc:2272] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.689900 7459 tablet_bootstrap.cc:654] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.694661 7462 tablet_bootstrap.cc:492] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:51.705674 7462 ts_tablet_manager.cc:1403] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.198s user 0.011s sys 0.001s
I20260317 21:18:51.672633 7467 raft_consensus.cc:740] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.708884 7462 raft_consensus.cc:359] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.709652 7462 raft_consensus.cc:385] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.711905 7462 raft_consensus.cc:740] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.712920 7467 consensus_queue.cc:260] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.718140 7462 consensus_queue.cc:260] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.719579 7483 ts_tablet_manager.cc:1916] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.725785 7467 ts_tablet_manager.cc:1434] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.066s user 0.009s sys 0.000s
I20260317 21:18:51.731794 7498 raft_consensus.cc:493] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:51.740718 7498 raft_consensus.cc:515] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.742599 7467 tablet_bootstrap.cc:492] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:51.743165 7498 leader_election.cc:290] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:51.746315 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019d90f6d70e46b087ea3c5ffd7cbd28" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:51.747603 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019d90f6d70e46b087ea3c5ffd7cbd28" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:51.748351 7183 raft_consensus.cc:2468] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:18:51.751405 7261 leader_election.cc:304] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
W20260317 21:18:51.753656 7264 leader_election.cc:343] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.754606 7498 raft_consensus.cc:2804] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:51.755031 7498 raft_consensus.cc:493] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:51.755386 7498 raft_consensus.cc:3060] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:51.761148 7459 tablet_bootstrap.cc:492] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:51.761768 7459 ts_tablet_manager.cc:1403] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.190s user 0.010s sys 0.005s
I20260317 21:18:51.761847 6666 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ffede0d29ebf494f8eec15158eeea6f6" } modify_external_catalogs: true
I20260317 21:18:51.762609 6666 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ffede0d29ebf494f8eec15158eeea6f6" } modify_external_catalogs: true
I20260317 21:18:51.767728 7467 tablet_bootstrap.cc:654] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.771930 7459 raft_consensus.cc:359] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.772738 7459 raft_consensus.cc:385] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:51.772984 7459 raft_consensus.cc:740] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:51.777331 7477 ts_tablet_manager.cc:1929] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:51.777868 7477 log.cc:1199] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/31725534b2a9401e9a79e8a467ed5210
I20260317 21:18:51.734041 7462 ts_tablet_manager.cc:1434] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.028s user 0.003s sys 0.003s
I20260317 21:18:51.778219 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_47f5d94de1a74b849740517dc73d3213"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:51.780031 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47f5d94de1a74b849740517dc73d3213 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.778465 7459 consensus_queue.cc:260] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.779412 7477 ts_tablet_manager.cc:1950] T 31725534b2a9401e9a79e8a467ed5210 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:51.781160 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1fa1b34540b546c6b6fbd89908efbdb3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:51.782975 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fa1b34540b546c6b6fbd89908efbdb3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.784987 7459 ts_tablet_manager.cc:1434] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.023s user 0.007s sys 0.000s
I20260317 21:18:51.788064 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 31725534b2a9401e9a79e8a467ed5210 (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]) successfully deleted
I20260317 21:18:51.801303 7549 raft_consensus.cc:493] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:51.802409 7549 raft_consensus.cc:515] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.803681 6671 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 2 got 0
I20260317 21:18:51.804314 7459 tablet_bootstrap.cc:492] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:51.810984 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:51.811769 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 56495c7cf3964743b96ab885afdaa152 (DEFAULT_TABLE table=default.Table_b1d7ee6663fe432a8940df97de65d99b [id=251402e78beb409192f3bd43ff21a9e6]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:51.812363 7261 leader_election.cc:343] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:51.813226 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56495c7cf3964743b96ab885afdaa152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.815825 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 (DEFAULT_TABLE table=default.Table_9f4465b6fdf24369a96e3bf37cfd19d7 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.816949 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 478f55d4c3034be0b1ef336a173c55b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.818557 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:51.819943 7549 leader_election.cc:290] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:51.823065 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 (DEFAULT_TABLE table=default.Table_9f4465b6fdf24369a96e3bf37cfd19d7 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.823879 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 56495c7cf3964743b96ab885afdaa152 (DEFAULT_TABLE table=default.Table_b1d7ee6663fe432a8940df97de65d99b [id=251402e78beb409192f3bd43ff21a9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.824381 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 478f55d4c3034be0b1ef336a173c55b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.825240 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56495c7cf3964743b96ab885afdaa152. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:51.825281 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 8deb47bf1eee48a09d8fd6543e409ec3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8deb47bf1eee48a09d8fd6543e409ec3 already in progress: creating tablet
W20260317 21:18:51.825989 7264 leader_election.cc:343] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:51.826396 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 3360925bb5e2415db39a551800ca28d0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3360925bb5e2415db39a551800ca28d0 already in progress: creating tablet
I20260317 21:18:51.826572 7264 leader_election.cc:304] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:51.828433 7477 tablet_replica.cc:333] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:51.829329 7477 raft_consensus.cc:2243] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:51.837014 7459 tablet_bootstrap.cc:654] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.841061 7477 raft_consensus.cc:2272] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:51.851477 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 56495c7cf3964743b96ab885afdaa152 (DEFAULT_TABLE table=default.Table_b1d7ee6663fe432a8940df97de65d99b [id=251402e78beb409192f3bd43ff21a9e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.852754 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56495c7cf3964743b96ab885afdaa152. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.857692 7009 tablet_service.cc:1511] Processing CreateTablet for tablet b5a797b22cf34483b0061955da6b2634 (DEFAULT_TABLE table=default.Table_2f865cde34f646c6b0058233b3bbc3a6 [id=4a69de09e4be4630994811c7e432c315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.858912 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a797b22cf34483b0061955da6b2634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.862848 7549 raft_consensus.cc:2749] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:51.868966 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 (DEFAULT_TABLE table=default.Table_9f4465b6fdf24369a96e3bf37cfd19d7 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.869669 7498 raft_consensus.cc:515] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.870149 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 478f55d4c3034be0b1ef336a173c55b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.874106 6671 catalog_manager.cc:5671] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:51.801537 7462 tablet_bootstrap.cc:492] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:51.881640 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019d90f6d70e46b087ea3c5ffd7cbd28" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:51.881987 7498 leader_election.cc:290] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:51.882302 7038 raft_consensus.cc:3060] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:51.885444 7477 ts_tablet_manager.cc:1916] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:51.886633 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 (DEFAULT_TABLE table=default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.887779 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06877bd7155842b9a5ae4a23ea68ada0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.891644 7038 raft_consensus.cc:2468] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:18:51.898064 7264 leader_election.cc:304] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:51.898887 7306 tablet_service.cc:1511] Processing CreateTablet for tablet b5a797b22cf34483b0061955da6b2634 (DEFAULT_TABLE table=default.Table_2f865cde34f646c6b0058233b3bbc3a6 [id=4a69de09e4be4630994811c7e432c315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.900344 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a797b22cf34483b0061955da6b2634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.878748 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_5c0d07d92b2d4bd3bfefa18fc6803f7a" } new_table_name: "default.Table_cceee5358ae04955b56b04763e90c07a" modify_external_catalogs: true
I20260317 21:18:51.904014 6671 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 2 got 0
I20260317 21:18:51.905117 7498 raft_consensus.cc:2804] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:51.906836 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4493a733fd0a496885b7745fcfc6cf2e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:51.908414 7498 raft_consensus.cc:697] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:18:51.909435 7498 consensus_queue.cc:237] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:51.911120 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4493a733fd0a496885b7745fcfc6cf2e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.879866 6668 master_service.cc:954] ReplaceTablet: received request to replace tablet 3360925bb5e2415db39a551800ca28d0 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:51.889531 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_47f5d94de1a74b849740517dc73d3213" } new_table_name: "default.Table_afb9c6074ee54f86a4d029cf8cdbb825" modify_external_catalogs: true
I20260317 21:18:51.901103 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 99b954ac41134300823458a7a25ef738
I20260317 21:18:51.919806 7156 tablet_service.cc:1511] Processing CreateTablet for tablet b5a797b22cf34483b0061955da6b2634 (DEFAULT_TABLE table=default.Table_2f865cde34f646c6b0058233b3bbc3a6 [id=4a69de09e4be4630994811c7e432c315]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.920212 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "019d90f6d70e46b087ea3c5ffd7cbd28" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:51.920838 7183 raft_consensus.cc:3060] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:51.921191 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5a797b22cf34483b0061955da6b2634. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.924001 7157 tablet_service.cc:1558] Processing DeleteTablet for tablet 99b954ac41134300823458a7a25ef738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:51.926826 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d50638a040344b81ad504d963feb3a4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:51.929549 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d50638a040344b81ad504d963feb3a4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:51.932030 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 99b954ac41134300823458a7a25ef738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:51.934051 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 99b954ac41134300823458a7a25ef738 because the tablet was not found. No further retry: Not found: Tablet not found: 99b954ac41134300823458a7a25ef738
I20260317 21:18:51.936039 7157 tablet_service.cc:1511] Processing CreateTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 (DEFAULT_TABLE table=default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.937237 7157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06877bd7155842b9a5ae4a23ea68ada0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.941781 7462 tablet_bootstrap.cc:654] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:51.942099 7159 tablet_service.cc:1511] Processing CreateTablet for tablet c1e672ca556e400ab05763abcd49ad77 (DEFAULT_TABLE table=default.Table_133ff1a27c134c24b1d11b25918e902d [id=57272480ab6a48ce8126d81d700e3833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.943297 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e672ca556e400ab05763abcd49ad77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.950242 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet 99b954ac41134300823458a7a25ef738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:51.951996 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 99b954ac41134300823458a7a25ef738 (DEFAULT_TABLE table=default.Table_93723f1459114d9ba82a21cee9158b3d [id=d079fd8ed7aa48ad9420c7cf7419c36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.952464 7017 tablet_service.cc:1511] Processing CreateTablet for tablet c1e672ca556e400ab05763abcd49ad77 (DEFAULT_TABLE table=default.Table_133ff1a27c134c24b1d11b25918e902d [id=57272480ab6a48ce8126d81d700e3833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.953233 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b954ac41134300823458a7a25ef738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.954497 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e672ca556e400ab05763abcd49ad77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.962029 7483 ts_tablet_manager.cc:1929] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:51.962473 7483 log.cc:1199] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/6343bd5fc6a44ac3976cbbed8fab26c7
I20260317 21:18:51.963157 7483 ts_tablet_manager.cc:1950] T 6343bd5fc6a44ac3976cbbed8fab26c7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:51.968021 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 99b954ac41134300823458a7a25ef738 (DEFAULT_TABLE table=default.Table_93723f1459114d9ba82a21cee9158b3d [id=d079fd8ed7aa48ad9420c7cf7419c36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:51.969726 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b954ac41134300823458a7a25ef738. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:51.981060 7459 tablet_bootstrap.cc:492] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:51.981596 7459 ts_tablet_manager.cc:1403] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.177s user 0.011s sys 0.020s
I20260317 21:18:51.983935 7459 raft_consensus.cc:359] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:51.993219 7183 raft_consensus.cc:2468] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:18:52.003722 7477 ts_tablet_manager.cc:1929] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.004256 7477 log.cc:1199] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/fc0fe1bcaeb04f7eb157172f9c216d56
I20260317 21:18:52.011417 7477 ts_tablet_manager.cc:1950] T fc0fe1bcaeb04f7eb157172f9c216d56 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:52.022889 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet fc0fe1bcaeb04f7eb157172f9c216d56 (table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]) successfully deleted
W20260317 21:18:52.026337 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 33f4e1fbb0d24813b0758cae9477f006 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 33f4e1fbb0d24813b0758cae9477f006 already in progress: creating tablet
I20260317 21:18:52.027604 6668 master_service.cc:954] ReplaceTablet: received request to replace tablet 8deb47bf1eee48a09d8fd6543e409ec3 from {username='slave'} at 127.0.0.1:39038
W20260317 21:18:52.028970 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6343bd5fc6a44ac3976cbbed8fab26c7 already in progress: creating tablet
I20260317 21:18:52.029695 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 (DEFAULT_TABLE table=default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.029999 7305 tablet_service.cc:1511] Processing CreateTablet for tablet c1e672ca556e400ab05763abcd49ad77 (DEFAULT_TABLE table=default.Table_133ff1a27c134c24b1d11b25918e902d [id=57272480ab6a48ce8126d81d700e3833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.030848 7462 tablet_bootstrap.cc:492] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:52.030937 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06877bd7155842b9a5ae4a23ea68ada0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.031332 7462 ts_tablet_manager.cc:1403] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.230s user 0.008s sys 0.003s
I20260317 21:18:52.031616 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e672ca556e400ab05763abcd49ad77. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.043237 7459 raft_consensus.cc:385] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.043594 7459 raft_consensus.cc:740] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.044251 7459 consensus_queue.cc:260] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.045689 7483 tablet_replica.cc:333] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:52.047945 7459 ts_tablet_manager.cc:1434] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.066s user 0.006s sys 0.000s
I20260317 21:18:52.053004 7498 raft_consensus.cc:493] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:52.053599 7498 raft_consensus.cc:515] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.058323 7467 tablet_bootstrap.cc:492] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:52.058796 7467 ts_tablet_manager.cc:1403] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.332s user 0.020s sys 0.019s
I20260317 21:18:52.058228 7462 raft_consensus.cc:359] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.059398 7462 raft_consensus.cc:385] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.059684 7462 raft_consensus.cc:740] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.060738 7483 raft_consensus.cc:2243] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.061214 7483 raft_consensus.cc:2272] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.061626 7467 raft_consensus.cc:359] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.062372 7467 raft_consensus.cc:385] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.062660 7467 raft_consensus.cc:740] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.063537 7467 consensus_queue.cc:260] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.068828 7483 ts_tablet_manager.cc:1916] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.068724 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 99b954ac41134300823458a7a25ef738 (DEFAULT_TABLE table=default.Table_93723f1459114d9ba82a21cee9158b3d [id=d079fd8ed7aa48ad9420c7cf7419c36e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.070135 7467 ts_tablet_manager.cc:1434] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.011s user 0.008s sys 0.000s
I20260317 21:18:52.072032 7462 consensus_queue.cc:260] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.071026 7467 tablet_bootstrap.cc:492] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:52.073901 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 6343bd5fc6a44ac3976cbbed8fab26c7 (table default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a]) successfully deleted
I20260317 21:18:52.078290 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet be2c469c792047d88fcf2ff9d53b7832
I20260317 21:18:52.080892 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b42fa238590b47d5bc0bd43dd3a313f2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:52.084228 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:52.085009 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99b954ac41134300823458a7a25ef738. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:52.085330 7264 leader_election.cc:343] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:52.085690 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 99b954ac41134300823458a7a25ef738 because the tablet was not found. No further retry: Not found: Tablet not found: 99b954ac41134300823458a7a25ef738
I20260317 21:18:52.093333 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_db1ed81759144b46920b2ce9dc6d912d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:52.095408 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db1ed81759144b46920b2ce9dc6d912d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:52.091717 6675 catalog_manager.cc:5671] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67). New cstate: current_term: 1 leader_uuid: "2f539d1a4a584b8ca83928be49369bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } }
W20260317 21:18:52.098232 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet be2c469c792047d88fcf2ff9d53b7832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be2c469c792047d88fcf2ff9d53b7832 already in progress: creating tablet
I20260317 21:18:52.101835 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:52.104561 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet be2c469c792047d88fcf2ff9d53b7832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be2c469c792047d88fcf2ff9d53b7832 already in progress: creating tablet
I20260317 21:18:52.116626 7459 tablet_bootstrap.cc:492] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:52.121133 7462 ts_tablet_manager.cc:1434] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.089s user 0.004s sys 0.004s
I20260317 21:18:52.121068 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b42fa238590b47d5bc0bd43dd3a313f2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:52.122512 7183 raft_consensus.cc:2468] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:18:52.123680 7261 leader_election.cc:304] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:52.124730 7549 raft_consensus.cc:2804] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:52.125094 7549 raft_consensus.cc:493] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:52.125413 7549 raft_consensus.cc:3060] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:52.129959 7498 leader_election.cc:290] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:52.131078 7459 tablet_bootstrap.cc:654] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.133671 7467 tablet_bootstrap.cc:654] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.136281 7549 raft_consensus.cc:515] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.140779 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.142063 7549 leader_election.cc:290] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:52.143057 7462 tablet_bootstrap.cc:492] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:52.144263 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b42fa238590b47d5bc0bd43dd3a313f2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:52.144245 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b42fa238590b47d5bc0bd43dd3a313f2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:52.145117 7183 raft_consensus.cc:3060] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:52.149017 7264 leader_election.cc:343] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:52.150192 7459 tablet_bootstrap.cc:492] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:52.150722 7459 ts_tablet_manager.cc:1403] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.034s user 0.008s sys 0.006s
I20260317 21:18:52.153434 7459 raft_consensus.cc:359] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.154183 7459 raft_consensus.cc:385] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.154487 7459 raft_consensus.cc:740] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.158692 7459 consensus_queue.cc:260] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.165663 7459 ts_tablet_manager.cc:1434] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.015s user 0.007s sys 0.001s
W20260317 21:18:52.167165 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 53dd9f8a1ccc47f884bfba99781aed16 has no consensus state
I20260317 21:18:52.167639 6668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b1d7ee6663fe432a8940df97de65d99b" } modify_external_catalogs: true
I20260317 21:18:52.168260 6668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b1d7ee6663fe432a8940df97de65d99b" } modify_external_catalogs: true
I20260317 21:18:52.169162 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:52.171052 7478 tablet_replica.cc:333] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:52.171957 7478 raft_consensus.cc:2243] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.172500 7478 raft_consensus.cc:2272] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.173306 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_8cba5340b2044350864c86b46e1a9488"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:52.175135 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cba5340b2044350864c86b46e1a9488 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:52.175437 7478 ts_tablet_manager.cc:1916] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:18:52.177835 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 019d90f6d70e46b087ea3c5ffd7cbd28 consensus state has no leader
I20260317 21:18:52.181509 7467 tablet_bootstrap.cc:492] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:52.181967 7467 ts_tablet_manager.cc:1403] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.111s user 0.007s sys 0.004s
I20260317 21:18:52.184346 7467 raft_consensus.cc:359] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.185074 7467 raft_consensus.cc:385] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.185310 7467 raft_consensus.cc:740] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.186775 6672 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954]). Expected version 3 got 0
I20260317 21:18:52.189472 7459 tablet_bootstrap.cc:492] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:52.190783 7462 tablet_bootstrap.cc:654] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.207263 7459 tablet_bootstrap.cc:654] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.209096 7467 consensus_queue.cc:260] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.217748 7467 ts_tablet_manager.cc:1434] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.036s user 0.006s sys 0.000s
W20260317 21:18:52.223896 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 83655.7us
I20260317 21:18:52.234629 7483 ts_tablet_manager.cc:1929] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.235028 7483 log.cc:1199] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/3360925bb5e2415db39a551800ca28d0
I20260317 21:18:52.235636 7483 ts_tablet_manager.cc:1950] T 3360925bb5e2415db39a551800ca28d0 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:52.240568 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 2 got 0
I20260317 21:18:52.253793 7183 raft_consensus.cc:2468] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:18:52.255576 7261 leader_election.cc:304] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:52.256605 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7377c42c199b408daa7f7e730ec5db06) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:52.256824 7549 raft_consensus.cc:2804] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:52.257262 7549 raft_consensus.cc:697] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:18:52.258035 7549 consensus_queue.cc:237] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.268911 7478 ts_tablet_manager.cc:1929] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.269492 7478 log.cc:1199] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:52.270454 7478 ts_tablet_manager.cc:1950] T 3ef4a015f5c14ee6afd045a87eaa6a89 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:52.226128 7467 tablet_bootstrap.cc:492] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:52.276499 7462 tablet_bootstrap.cc:492] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:52.280090 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 3360925bb5e2415db39a551800ca28d0 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]) successfully deleted
I20260317 21:18:52.282245 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 3ef4a015f5c14ee6afd045a87eaa6a89 (table default.Table_f3adbc8bf1994597bf7be6a6792ebf5a [id=16fb03a1fcfb4eff833f8ff6292c0b21]) successfully deleted
I20260317 21:18:52.287179 7478 tablet_replica.cc:333] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:52.287927 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet b42fa238590b47d5bc0bd43dd3a313f2 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:52.288297 7462 ts_tablet_manager.cc:1403] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.145s user 0.009s sys 0.007s
I20260317 21:18:52.291244 7462 raft_consensus.cc:359] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.291913 7462 raft_consensus.cc:385] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.292166 7462 raft_consensus.cc:740] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:18:52.292434 6662 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 60f3d32e1f7649fa92861f51134cf000 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 60f3d32e1f7649fa92861f51134cf000 has no consensus state
I20260317 21:18:52.287853 7483 tablet_replica.cc:333] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:52.293195 7483 raft_consensus.cc:2243] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.293615 7483 raft_consensus.cc:2272] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.295306 7483 ts_tablet_manager.cc:1916] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.302556 7459 tablet_bootstrap.cc:492] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:52.302996 7459 ts_tablet_manager.cc:1403] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.114s user 0.012s sys 0.005s
I20260317 21:18:52.305390 7467 tablet_bootstrap.cc:654] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.308220 7459 raft_consensus.cc:359] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.309302 7459 raft_consensus.cc:385] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.309413 7478 raft_consensus.cc:2243] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.309585 7459 raft_consensus.cc:740] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.309887 7478 raft_consensus.cc:2272] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.309809 6667 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 2 got 0
I20260317 21:18:52.312623 7459 consensus_queue.cc:260] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.313978 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 7377c42c199b408daa7f7e730ec5db06) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:52.314993 7477 tablet_replica.cc:333] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:52.318020 7478 ts_tablet_manager.cc:1916] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.322451 7462 consensus_queue.cc:260] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.326066 7477 raft_consensus.cc:2243] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.326589 7477 raft_consensus.cc:2272] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.326673 6668 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 56495c7cf3964743b96ab885afdaa152
I20260317 21:18:52.328833 7477 ts_tablet_manager.cc:1916] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.329058 7459 ts_tablet_manager.cc:1434] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.026s user 0.008s sys 0.000s
I20260317 21:18:52.333421 7528 raft_consensus.cc:493] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:52.334031 7528 raft_consensus.cc:515] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.336071 7528 leader_election.cc:290] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:52.338362 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:52.338973 7462 ts_tablet_manager.cc:1434] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.050s user 0.006s sys 0.000s
I20260317 21:18:52.339219 7329 raft_consensus.cc:2393] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:18:52.339859 7462 tablet_bootstrap.cc:492] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:52.346524 7459 tablet_bootstrap.cc:492] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:52.347258 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f41f18434a744b03b6ff53f3d7760e55" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:52.349150 6971 leader_election.cc:304] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:52.350850 7528 raft_consensus.cc:3060] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:52.361931 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 56495c7cf3964743b96ab885afdaa152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:52.363766 7157 tablet_service.cc:1558] Processing DeleteTablet for tablet 56495c7cf3964743b96ab885afdaa152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:52.368291 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 51754.2us
I20260317 21:18:52.369781 7467 tablet_bootstrap.cc:492] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:52.370229 7467 ts_tablet_manager.cc:1403] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.005s
I20260317 21:18:52.372761 7467 raft_consensus.cc:359] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.373461 7467 raft_consensus.cc:385] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.373731 7467 raft_consensus.cc:740] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.374742 7528 raft_consensus.cc:2749] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:52.376328 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 56495c7cf3964743b96ab885afdaa152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.377848 7467 consensus_queue.cc:260] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.377435 7459 tablet_bootstrap.cc:654] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.397526 7462 tablet_bootstrap.cc:654] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.400116 7329 raft_consensus.cc:1275] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Refusing update from remote peer c0ef92fef382401a9564c17e0c223f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:18:52.402688 7038 raft_consensus.cc:1275] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Refusing update from remote peer c0ef92fef382401a9564c17e0c223f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:18:52.404760 7538 consensus_queue.cc:1048] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:18:52.406047 7467 ts_tablet_manager.cc:1434] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.031s user 0.004s sys 0.003s
I20260317 21:18:52.406654 7538 consensus_queue.cc:1048] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:18:52.407166 7467 tablet_bootstrap.cc:492] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:52.408262 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_eec7ee55d8bf4e9baa04680fd31ac46c" } modify_external_catalogs: true
I20260317 21:18:52.409111 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_eec7ee55d8bf4e9baa04680fd31ac46c" } modify_external_catalogs: true
I20260317 21:18:52.411013 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet f0d55550bf914019b83e83fa4ffcb64b from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:52.426695 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 3360925bb5e2415db39a551800ca28d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 60f3d32e1f7649fa92861f51134cf000) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:52.440855 6671 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f025e5d87c5f4a119d195af00e00dd38" } new_table_name: "default.Table_32efbd6caca74d13a6d567c3f470d9a1" modify_external_catalogs: true
I20260317 21:18:52.441704 7477 ts_tablet_manager.cc:1929] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.442091 7477 log.cc:1199] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/6343bd5fc6a44ac3976cbbed8fab26c7
I20260317 21:18:52.442935 7477 ts_tablet_manager.cc:1950] T 6343bd5fc6a44ac3976cbbed8fab26c7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:52.447760 7483 ts_tablet_manager.cc:1929] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.448163 7483 log.cc:1199] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:52.457286 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_312acc8080704bcab4d8d459b8634d8a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:52.459625 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_312acc8080704bcab4d8d459b8634d8a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:52.460359 7483 ts_tablet_manager.cc:1950] T 33f4e1fbb0d24813b0758cae9477f006 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:52.467173 7467 tablet_bootstrap.cc:654] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.467417 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 33f4e1fbb0d24813b0758cae9477f006 (table default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]) successfully deleted
I20260317 21:18:52.467182 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1fa1b34540b546c6b6fbd89908efbdb3" } new_table_name: "default.Table_cae455ad5e214610bcfbeff2d7054eff" modify_external_catalogs: true
I20260317 21:18:52.475755 7459 tablet_bootstrap.cc:492] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:52.476228 7459 ts_tablet_manager.cc:1403] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.000s
I20260317 21:18:52.487296 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet b42fa238590b47d5bc0bd43dd3a313f2
I20260317 21:18:52.487926 7538 raft_consensus.cc:493] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:52.488399 7538 raft_consensus.cc:515] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.490296 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 with delete_type TABLET_DATA_DELETED (replaced by tablet d2f7c1058f8245428f38fe3245621b75) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:52.491184 7538 leader_election.cc:290] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:52.492329 6672 catalog_manager.cc:6741] ReplaceTablet: tablet b42fa238590b47d5bc0bd43dd3a313f2 deleted and replaced by tablet d2f7c1058f8245428f38fe3245621b75
I20260317 21:18:52.493583 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 with delete_type TABLET_DATA_DELETED (replaced by tablet d2f7c1058f8245428f38fe3245621b75) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.495239 7183 raft_consensus.cc:1275] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Refusing update from remote peer 2f539d1a4a584b8ca83928be49369bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:18:52.497216 7549 consensus_queue.cc:1048] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:18:52.498064 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 6343bd5fc6a44ac3976cbbed8fab26c7 because the tablet was not found. No further retry: Not found: Tablet not found: 6343bd5fc6a44ac3976cbbed8fab26c7
W20260317 21:18:52.500494 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 99b954ac41134300823458a7a25ef738 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 99b954ac41134300823458a7a25ef738 already in progress: creating tablet
I20260317 21:18:52.500612 7036 raft_consensus.cc:1275] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Refusing update from remote peer 2f539d1a4a584b8ca83928be49369bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:18:52.501677 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3360925bb5e2415db39a551800ca28d0" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:52.506047 7120 leader_election.cc:343] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:52.505192 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_544a3d9c42534ca38052e43d62ca136a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:52.506820 7549 consensus_queue.cc:1048] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:52.507470 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 with delete_type TABLET_DATA_DELETED (replaced by tablet d2f7c1058f8245428f38fe3245621b75) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:52.508878 7477 tablet_replica.cc:333] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:52.509728 7477 raft_consensus.cc:2243] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.510303 7477 raft_consensus.cc:2272] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:18:52.510445 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet be2c469c792047d88fcf2ff9d53b7832 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet be2c469c792047d88fcf2ff9d53b7832 already in progress: creating tablet
W20260317 21:18:52.513321 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 56495c7cf3964743b96ab885afdaa152 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56495c7cf3964743b96ab885afdaa152 already in progress: creating tablet
I20260317 21:18:52.490551 7478 ts_tablet_manager.cc:1929] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.516243 7478 log.cc:1199] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/6343bd5fc6a44ac3976cbbed8fab26c7
W20260317 21:18:52.521064 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_544a3d9c42534ca38052e43d62ca136a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:52.523825 7462 tablet_bootstrap.cc:492] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:52.526616 7467 tablet_bootstrap.cc:492] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:52.527324 7467 ts_tablet_manager.cc:1403] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.120s user 0.012s sys 0.005s
I20260317 21:18:52.527266 7477 ts_tablet_manager.cc:1916] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.521510 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 6343bd5fc6a44ac3976cbbed8fab26c7 (table default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a]) successfully deleted
W20260317 21:18:52.528546 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 56495c7cf3964743b96ab885afdaa152 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56495c7cf3964743b96ab885afdaa152 already in progress: creating tablet
I20260317 21:18:52.517402 7478 ts_tablet_manager.cc:1950] T 6343bd5fc6a44ac3976cbbed8fab26c7 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:52.534682 7459 raft_consensus.cc:359] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.535449 7459 raft_consensus.cc:385] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.535856 7459 raft_consensus.cc:740] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.538410 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet f0d55550bf914019b83e83fa4ffcb64b
I20260317 21:18:52.539968 6666 catalog_manager.cc:6741] ReplaceTablet: tablet f0d55550bf914019b83e83fa4ffcb64b deleted and replaced by tablet e4522181cc434203ac14c9b07587fec8
I20260317 21:18:52.545010 7462 ts_tablet_manager.cc:1403] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.205s user 0.012s sys 0.009s
I20260317 21:18:52.547726 7462 raft_consensus.cc:359] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:52.525147 7483 tablet_replica.cc:333] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
W20260317 21:18:52.550200 6671 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 163cc8c04ca94a269c7ae2efaa162525 has no consensus state
I20260317 21:18:52.555554 7467 raft_consensus.cc:359] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.556392 7467 raft_consensus.cc:385] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.556699 7467 raft_consensus.cc:740] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.557391 7467 consensus_queue.cc:260] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.558980 7483 raft_consensus.cc:2243] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:52.559804 7483 pending_rounds.cc:70] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Trying to abort 1 pending ops.
I20260317 21:18:52.560161 7483 pending_rounds.cc:77] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412432937570304 op_type: NO_OP noop_request { }
I20260317 21:18:52.560787 7483 raft_consensus.cc:2889] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:52.556105 7459 consensus_queue.cc:260] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.561271 7483 raft_consensus.cc:2272] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.562140 7528 raft_consensus.cc:493] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:52.562933 7528 raft_consensus.cc:515] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.548435 7462 raft_consensus.cc:385] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.564163 7462 raft_consensus.cc:740] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.567698 7462 consensus_queue.cc:260] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:52.571156 7483 ts_tablet_manager.cc:1916] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.582631 7563 mvcc.cc:204] Tried to move back new op lower bound from 7265412433717428224 to 7265412431539175424. Current Snapshot: MvccSnapshot[applied={T|T < 7265412433717428224}]
I20260317 21:18:52.590247 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:52.591632 7462 ts_tablet_manager.cc:1434] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260317 21:18:52.591289 7329 raft_consensus.cc:2468] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:52.592414 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet f0d55550bf914019b83e83fa4ffcb64b with delete_type TABLET_DATA_DELETED (replaced by tablet e4522181cc434203ac14c9b07587fec8) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:52.592829 6973 leader_election.cc:304] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:52.598328 7576 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:18:52.591317 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet f0d55550bf914019b83e83fa4ffcb64b with delete_type TABLET_DATA_DELETED (replaced by tablet e4522181cc434203ac14c9b07587fec8) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.622290 7528 leader_election.cc:290] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:52.622570 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3a13373a17d3475095ae98c63ab87fea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:52.625593 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bb6c327aff234752a4f6930f24dc07b9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:52.626120 7528 raft_consensus.cc:2804] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:52.626452 7528 raft_consensus.cc:493] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:52.626753 7528 raft_consensus.cc:3060] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:52.628161 6674 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_db1ed81759144b46920b2ce9dc6d912d" } modify_external_catalogs: true
I20260317 21:18:52.629144 6674 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_db1ed81759144b46920b2ce9dc6d912d" } modify_external_catalogs: true
I20260317 21:18:52.614284 7462 tablet_bootstrap.cc:492] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:18:52.652725 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a13373a17d3475095ae98c63ab87fea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:52.656898 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb6c327aff234752a4f6930f24dc07b9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:52.657511 7119 leader_election.cc:343] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3360925bb5e2415db39a551800ca28d0
I20260317 21:18:52.658191 7119 leader_election.cc:304] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:52.666451 7580 raft_consensus.cc:493] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:52.667882 7580 raft_consensus.cc:515] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:52.680672 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d59deea79cfb4a83831bb0227797c690 has no consensus state
I20260317 21:18:52.682355 6674 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 2630e8f621cb477da2c7e33f3c082432
I20260317 21:18:52.700337 7483 ts_tablet_manager.cc:1929] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:52.676990 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet f0d55550bf914019b83e83fa4ffcb64b with delete_type TABLET_DATA_DELETED (replaced by tablet e4522181cc434203ac14c9b07587fec8) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:52.679889 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:52.699031 7478 tablet_replica.cc:333] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:52.708608 7467 ts_tablet_manager.cc:1434] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.180s user 0.006s sys 0.006s
W20260317 21:18:52.720306 7572 log.cc:927] Time spent T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Append to log took a long time: real 0.159s user 0.001s sys 0.000s
I20260317 21:18:52.718931 7583 raft_consensus.cc:2749] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:52.753005 7467 tablet_bootstrap.cc:492] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:52.756601 7478 raft_consensus.cc:2243] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.757385 7478 raft_consensus.cc:2272] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.764850 7483 log.cc:1199] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/b42fa238590b47d5bc0bd43dd3a313f2
W20260317 21:18:52.681504 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 88831.3us
I20260317 21:18:52.766563 7483 ts_tablet_manager.cc:1950] T b42fa238590b47d5bc0bd43dd3a313f2 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:52.774104 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet b42fa238590b47d5bc0bd43dd3a313f2 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:18:52.759466 7459 ts_tablet_manager.cc:1434] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.283s user 0.014s sys 0.001s
I20260317 21:18:52.785331 7459 tablet_bootstrap.cc:492] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:52.791399 7478 ts_tablet_manager.cc:1916] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:52.793583 7584 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:18:52.815035 7184 raft_consensus.cc:2468] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:52.817687 7017 tablet_service.cc:1511] Processing CreateTablet for tablet d59deea79cfb4a83831bb0227797c690 (DEFAULT_TABLE table=default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.818867 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59deea79cfb4a83831bb0227797c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.821029 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_cceee5358ae04955b56b04763e90c07a [id=2ed77cec958e4b008989a60fb67a2adc]). Expected version 1 got 0
I20260317 21:18:52.821134 7528 raft_consensus.cc:515] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:52.748404 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 70319.6us
I20260317 21:18:52.823509 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 6343bd5fc6a44ac3976cbbed8fab26c7 (table default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a]) successfully deleted
I20260317 21:18:52.831655 7585 tablet.cc:1723] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:52.833148 7459 tablet_bootstrap.cc:654] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.836242 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:52.836951 7329 raft_consensus.cc:2468] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:52.837508 6674 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_cceee5358ae04955b56b04763e90c07a [id=2ed77cec958e4b008989a60fb67a2adc]). Expected version 1 got 0
I20260317 21:18:52.837965 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 (DEFAULT_TABLE table=default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.838198 7119 leader_election.cc:304] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:52.839148 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba0874a8cd746e9a33596ea252b3b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.841161 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:52.841717 7329 raft_consensus.cc:3060] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:52.842031 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:52.842926 7184 raft_consensus.cc:3060] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:52.850950 7329 raft_consensus.cc:2468] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:52.851497 7528 leader_election.cc:290] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:52.853153 6973 leader_election.cc:304] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:52.854086 7528 raft_consensus.cc:2804] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:52.855320 7528 raft_consensus.cc:697] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:52.858098 7528 consensus_queue.cc:237] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:52.868155 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 84646.7us
I20260317 21:18:52.871300 7308 tablet_service.cc:1511] Processing CreateTablet for tablet d59deea79cfb4a83831bb0227797c690 (DEFAULT_TABLE table=default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.871821 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet bc93b8e8da49471493369deef1a502f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.872359 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59deea79cfb4a83831bb0227797c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.872494 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 with delete_type TABLET_DATA_DELETED (replaced by tablet d2f7c1058f8245428f38fe3245621b75) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.874902 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:52.875473 7459 tablet_bootstrap.cc:492] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:52.875895 7459 ts_tablet_manager.cc:1403] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.091s user 0.011s sys 0.000s
W20260317 21:18:52.887183 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 56495c7cf3964743b96ab885afdaa152 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 56495c7cf3964743b96ab885afdaa152 already in progress: creating tablet
I20260317 21:18:52.889021 7576 tablet.cc:1723] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:52.895905 7565 mvcc.cc:204] Tried to move back new op lower bound from 7265412433123049472 to 7265412430456582144. Current Snapshot: MvccSnapshot[applied={T|T < 7265412433123049472}]
I20260317 21:18:52.907873 7462 tablet_bootstrap.cc:654] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.913273 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "742b03ea3048466da3c9ea5ed9010b27" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:52.916765 7467 tablet_bootstrap.cc:654] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:52.906420 7589 tablet.cc:1723] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:52.922098 7498 raft_consensus.cc:493] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:52.922576 7498 raft_consensus.cc:515] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.924546 7498 leader_election.cc:290] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:52.926477 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:52.927003 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_133ff1a27c134c24b1d11b25918e902d" } new_table_name: "default.Table_5f4dc71734d540319e6f0d19d4037ba8" modify_external_catalogs: true
I20260317 21:18:52.927367 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet 3360925bb5e2415db39a551800ca28d0 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:52.924177 7459 raft_consensus.cc:359] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:52.928498 7264 leader_election.cc:343] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:52.928946 7459 raft_consensus.cc:385] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:52.929224 7459 raft_consensus.cc:740] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:52.929016 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e717b96ed78c4b5d8aab97d49666ebea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:52.930783 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e717b96ed78c4b5d8aab97d49666ebea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:52.931502 7184 raft_consensus.cc:2468] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:52.933542 7459 consensus_queue.cc:260] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:52.937812 7009 tablet_service.cc:1511] Processing CreateTablet for tablet e1094068903840a8ab5d07c7fb0635bf (DEFAULT_TABLE table=default.Table_6dc2ff9e049443948de3753522e042f8 [id=93623d5786f645c7a1ad496e656caa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.938989 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1094068903840a8ab5d07c7fb0635bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.947311 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:52.932039 7583 raft_consensus.cc:2764] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20260317 21:18:52.949980 7261 leader_election.cc:343] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:52.950557 7261 leader_election.cc:304] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:52.952903 7549 raft_consensus.cc:2749] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:52.868520 7580 leader_election.cc:290] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:52.957142 7483 tablet_replica.cc:333] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:52.953572 7459 ts_tablet_manager.cc:1434] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.077s user 0.007s sys 0.000s
I20260317 21:18:52.957926 7483 raft_consensus.cc:2243] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:52.958536 7483 raft_consensus.cc:2272] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:52.961321 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet c1e672ca556e400ab05763abcd49ad77 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:52.965109 7477 ts_tablet_manager.cc:1929] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:52.965520 7477 log.cc:1199] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/b42fa238590b47d5bc0bd43dd3a313f2
I20260317 21:18:52.966010 7459 tablet_bootstrap.cc:492] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:52.966364 7477 ts_tablet_manager.cc:1950] T b42fa238590b47d5bc0bd43dd3a313f2 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:52.966687 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b88a46682c9e48a8a2ed67ad3a8f47f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:52.969084 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b88a46682c9e48a8a2ed67ad3a8f47f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:52.915011 7590 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:18:52.978399 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_cceee5358ae04955b56b04763e90c07a" } new_table_name: "default.Table_396735bd5352499580e29038e6317fd4" modify_external_catalogs: true
I20260317 21:18:52.981765 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet b42fa238590b47d5bc0bd43dd3a313f2 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
W20260317 21:18:52.984900 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet f0d55550bf914019b83e83fa4ffcb64b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f0d55550bf914019b83e83fa4ffcb64b already in progress: creating tablet
W20260317 21:18:52.991322 6662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111) [suppressed 9 similar messages]
I20260317 21:18:52.992470 7158 tablet_service.cc:1511] Processing CreateTablet for tablet d59deea79cfb4a83831bb0227797c690 (DEFAULT_TABLE table=default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.994666 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59deea79cfb4a83831bb0227797c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.996508 6672 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_7a69602e996b4ca6aa936f73b9b24c34" } modify_external_catalogs: true
I20260317 21:18:52.997185 6672 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_7a69602e996b4ca6aa936f73b9b24c34" } modify_external_catalogs: true
I20260317 21:18:52.998351 7157 tablet_service.cc:1511] Processing CreateTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 (DEFAULT_TABLE table=default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:52.999656 7157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba0874a8cd746e9a33596ea252b3b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.001706 7154 tablet_service.cc:1511] Processing CreateTablet for tablet e1094068903840a8ab5d07c7fb0635bf (DEFAULT_TABLE table=default.Table_6dc2ff9e049443948de3753522e042f8 [id=93623d5786f645c7a1ad496e656caa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.003257 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1094068903840a8ab5d07c7fb0635bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:52.996542 7459 tablet_bootstrap.cc:654] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.009052 7483 ts_tablet_manager.cc:1916] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.011543 7498 raft_consensus.cc:493] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:53.012253 7498 raft_consensus.cc:515] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.014916 7498 leader_election.cc:290] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:53.017292 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8deb47bf1eee48a09d8fd6543e409ec3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:53.017940 7184 raft_consensus.cc:2468] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:18:53.019424 7261 leader_election.cc:304] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:53.021076 7498 raft_consensus.cc:2804] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:53.021512 7498 raft_consensus.cc:493] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:53.022360 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8deb47bf1eee48a09d8fd6543e409ec3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:53.021912 7498 raft_consensus.cc:3060] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:53.023001 7036 raft_consensus.cc:2468] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:18:53.043103 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 (DEFAULT_TABLE table=default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.044159 7305 tablet_service.cc:1511] Processing CreateTablet for tablet e1094068903840a8ab5d07c7fb0635bf (DEFAULT_TABLE table=default.Table_6dc2ff9e049443948de3753522e042f8 [id=93623d5786f645c7a1ad496e656caa24]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.044200 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ba0874a8cd746e9a33596ea252b3b14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.049007 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1094068903840a8ab5d07c7fb0635bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.055096 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 (DEFAULT_TABLE table=default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.056295 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163cc8c04ca94a269c7ae2efaa162525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.070184 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 (DEFAULT_TABLE table=default.Table_afb9c6074ee54f86a4d029cf8cdbb825 [id=dd080d09b8324bcabbe022d42352fadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.071323 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53dd9f8a1ccc47f884bfba99781aed16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.080926 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 (DEFAULT_TABLE table=default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.082618 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163cc8c04ca94a269c7ae2efaa162525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.089519 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_638c2b3200334f309baef743312b646c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:53.091815 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_638c2b3200334f309baef743312b646c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:53.093606 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_ed089cf2f5554ed38d669ac4976ee78c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:53.095749 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ed089cf2f5554ed38d669ac4976ee78c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:53.105537 7459 tablet_bootstrap.cc:492] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:53.106006 7459 ts_tablet_manager.cc:1403] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.145s user 0.021s sys 0.001s
I20260317 21:18:53.118574 7459 raft_consensus.cc:359] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.119246 7459 raft_consensus.cc:385] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.119500 7459 raft_consensus.cc:740] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.123356 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 (DEFAULT_TABLE table=default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.124763 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 163cc8c04ca94a269c7ae2efaa162525. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.120129 7459 consensus_queue.cc:260] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.138429 7459 ts_tablet_manager.cc:1434] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260317 21:18:53.138335 7498 raft_consensus.cc:515] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.139849 7459 tablet_bootstrap.cc:492] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:53.142436 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8deb47bf1eee48a09d8fd6543e409ec3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:53.143126 7036 raft_consensus.cc:3060] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:53.146420 7467 tablet_bootstrap.cc:492] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:53.146860 7467 ts_tablet_manager.cc:1403] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.394s user 0.005s sys 0.016s
I20260317 21:18:53.151095 7467 raft_consensus.cc:359] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:53.152037 7467 raft_consensus.cc:385] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.152081 7156 tablet_service.cc:1511] Processing CreateTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 (DEFAULT_TABLE table=default.Table_afb9c6074ee54f86a4d029cf8cdbb825 [id=dd080d09b8324bcabbe022d42352fadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.154763 7478 ts_tablet_manager.cc:1929] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:53.155846 7478 log.cc:1199] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/3360925bb5e2415db39a551800ca28d0
I20260317 21:18:53.163187 7036 raft_consensus.cc:2468] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:18:53.164999 7264 leader_election.cc:304] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:53.165757 7549 raft_consensus.cc:2804] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:53.166986 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 (DEFAULT_TABLE table=default.Table_afb9c6074ee54f86a4d029cf8cdbb825 [id=dd080d09b8324bcabbe022d42352fadb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.167462 7528 raft_consensus.cc:493] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:53.168026 7528 raft_consensus.cc:515] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.168480 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53dd9f8a1ccc47f884bfba99781aed16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.169992 7528 leader_election.cc:290] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:53.170833 7462 tablet_bootstrap.cc:492] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:53.171207 7462 ts_tablet_manager.cc:1403] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.557s user 0.015s sys 0.006s
I20260317 21:18:53.171633 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:53.172917 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:53.173286 7014 tablet_service.cc:1511] Processing CreateTablet for tablet f749358db0f74ef78d9eee23b72bfe3e (DEFAULT_TABLE table=default.Table_49f59ddc9db844c5bcfb828a4735238b [id=944a00d9ea784873b8e346589b9d86b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.173483 7329 raft_consensus.cc:2468] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:53.174403 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f749358db0f74ef78d9eee23b72bfe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.174099 7462 raft_consensus.cc:359] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:53.174752 7462 raft_consensus.cc:385] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.175025 7462 raft_consensus.cc:740] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.175664 7462 consensus_queue.cc:260] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:53.176954 7478 ts_tablet_manager.cc:1950] T 3360925bb5e2415db39a551800ca28d0 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:53.186058 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 742b03ea3048466da3c9ea5ed9010b27
I20260317 21:18:53.190155 6675 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f] alter complete (version 2)
I20260317 21:18:53.202270 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 742b03ea3048466da3c9ea5ed9010b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:53.204563 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:53.209941 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 742b03ea3048466da3c9ea5ed9010b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:53.174836 6971 leader_election.cc:343] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:53.220757 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53dd9f8a1ccc47f884bfba99781aed16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.221535 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8deb47bf1eee48a09d8fd6543e409ec3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:53.222095 7184 raft_consensus.cc:3060] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:53.222963 7576 tablet.cc:1718] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Skipping requested alter to schema version 2, tablet already version 2
I20260317 21:18:53.230934 7184 raft_consensus.cc:2468] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:18:53.237154 7498 leader_election.cc:290] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:18:53.258961 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c1e672ca556e400ab05763abcd49ad77 consensus state has no leader
I20260317 21:18:53.261284 7559 mvcc.cc:204] Tried to move back new op lower bound from 7265412436775219200 to 7265412430456582144. Current Snapshot: MvccSnapshot[applied={T|T < 7265412436775219200}]
I20260317 21:18:53.262606 7585 tablet.cc:1718] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Skipping requested alter to schema version 2, tablet already version 2
I20260317 21:18:53.273469 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 2 got 0
I20260317 21:18:53.281544 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet c1e672ca556e400ab05763abcd49ad77
I20260317 21:18:53.283494 6666 catalog_manager.cc:6741] ReplaceTablet: tablet c1e672ca556e400ab05763abcd49ad77 deleted and replaced by tablet dc929bae186841d2aba674659e457301
I20260317 21:18:53.152474 7467 raft_consensus.cc:740] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.288096 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:53.290333 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:53.296041 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 742b03ea3048466da3c9ea5ed9010b27 consensus state has no leader
I20260317 21:18:53.298660 7467 consensus_queue.cc:260] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:53.302416 7155 tablet_service.cc:1511] Processing CreateTablet for tablet f749358db0f74ef78d9eee23b72bfe3e (DEFAULT_TABLE table=default.Table_49f59ddc9db844c5bcfb828a4735238b [id=944a00d9ea784873b8e346589b9d86b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.304252 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f749358db0f74ef78d9eee23b72bfe3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.312937 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 3360925bb5e2415db39a551800ca28d0 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]) successfully deleted
I20260317 21:18:53.326591 7308 tablet_service.cc:1511] Processing CreateTablet for tablet f749358db0f74ef78d9eee23b72bfe3e (DEFAULT_TABLE table=default.Table_49f59ddc9db844c5bcfb828a4735238b [id=944a00d9ea784873b8e346589b9d86b6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.329406 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f749358db0f74ef78d9eee23b72bfe3e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:53.335644 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 160832us
I20260317 21:18:53.338071 7467 ts_tablet_manager.cc:1434] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.191s user 0.009s sys 0.000s
I20260317 21:18:53.338986 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:53.349972 7459 tablet_bootstrap.cc:654] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.352837 7478 tablet_replica.cc:333] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:53.359973 7478 raft_consensus.cc:2243] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:53.360667 7478 raft_consensus.cc:2272] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.362279 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 742b03ea3048466da3c9ea5ed9010b27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:53.362648 7478 ts_tablet_manager.cc:1916] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.362829 7549 raft_consensus.cc:697] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:18:53.363721 6973 leader_election.cc:304] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:53.363811 7549 consensus_queue.cc:237] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.364833 7528 raft_consensus.cc:2804] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:53.365159 7528 raft_consensus.cc:493] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:53.365481 7528 raft_consensus.cc:3060] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:53.373222 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 60f3d32e1f7649fa92861f51134cf000 (DEFAULT_TABLE table=default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.374339 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 60f3d32e1f7649fa92861f51134cf000 (DEFAULT_TABLE table=default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.375739 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f3d32e1f7649fa92861f51134cf000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.376339 7467 tablet_bootstrap.cc:492] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:53.381314 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f3d32e1f7649fa92861f51134cf000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.381983 7477 tablet_replica.cc:333] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:53.382674 7477 raft_consensus.cc:2243] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:53.383447 7477 pending_rounds.cc:70] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:18:53.383728 7477 pending_rounds.cc:77] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412435439169536 op_type: NO_OP noop_request { }
I20260317 21:18:53.384114 7477 raft_consensus.cc:2889] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:53.384439 7477 raft_consensus.cc:2272] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:18:53.386411 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet c1e672ca556e400ab05763abcd49ad77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1e672ca556e400ab05763abcd49ad77 already in progress: creating tablet
I20260317 21:18:53.386601 7462 ts_tablet_manager.cc:1434] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.215s user 0.006s sys 0.000s
I20260317 21:18:53.391884 7462 tablet_bootstrap.cc:492] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:53.416049 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 60f3d32e1f7649fa92861f51134cf000 (DEFAULT_TABLE table=default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:53.427843 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_2fab17210bfa496ba34bb9580bee9d79" } modify_external_catalogs: true
I20260317 21:18:53.428524 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_2fab17210bfa496ba34bb9580bee9d79" } modify_external_catalogs: true
I20260317 21:18:53.435318 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60f3d32e1f7649fa92861f51134cf000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:53.447207 7477 ts_tablet_manager.cc:1916] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.449465 7467 tablet_bootstrap.cc:654] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.461946 7462 tablet_bootstrap.cc:654] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:53.449518 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 273869us
I20260317 21:18:53.498521 7462 tablet_bootstrap.cc:492] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:53.498977 7462 ts_tablet_manager.cc:1403] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.107s user 0.008s sys 0.004s
I20260317 21:18:53.502928 7483 ts_tablet_manager.cc:1929] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:53.503324 7483 log.cc:1199] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/f0d55550bf914019b83e83fa4ffcb64b
I20260317 21:18:53.503983 7483 ts_tablet_manager.cc:1950] T f0d55550bf914019b83e83fa4ffcb64b P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:53.509614 7528 raft_consensus.cc:515] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.514909 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_312acc8080704bcab4d8d459b8634d8a" } new_table_name: "default.Table_2e5acf8292c248929fd6beead0350ea4" modify_external_catalogs: true
I20260317 21:18:53.517763 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4aa5ce0a35044d249a974aee7ccb904b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:53.519531 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4aa5ce0a35044d249a974aee7ccb904b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:53.524464 7483 tablet_replica.cc:333] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:53.525264 7483 raft_consensus.cc:2243] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:53.525719 7483 raft_consensus.cc:2272] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.527386 7483 ts_tablet_manager.cc:1916] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.530962 7462 raft_consensus.cc:359] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.531664 7462 raft_consensus.cc:385] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.531936 7462 raft_consensus.cc:740] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.545536 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 60f3d32e1f7649fa92861f51134cf000
I20260317 21:18:53.545140 7462 consensus_queue.cc:260] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.563099 7462 ts_tablet_manager.cc:1434] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.064s user 0.006s sys 0.001s
I20260317 21:18:53.559131 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet b42fa238590b47d5bc0bd43dd3a313f2 with delete_type TABLET_DATA_DELETED (replaced by tablet d2f7c1058f8245428f38fe3245621b75) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:53.570380 7462 tablet_bootstrap.cc:492] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:53.572077 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 60f3d32e1f7649fa92861f51134cf000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:53.587899 7462 tablet_bootstrap.cc:654] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:53.601970 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 67671us
I20260317 21:18:53.603030 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet f0d55550bf914019b83e83fa4ffcb64b (table default.Table_5bd5abc0a4a64c408b1bae33441413d4 [id=577b3aea931d4e518d588b81917b49c8]) successfully deleted
I20260317 21:18:53.627776 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:18:53.629843 6971 leader_election.cc:343] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:53.630844 7528 leader_election.cc:290] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:53.632586 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
W20260317 21:18:53.634907 6973 leader_election.cc:343] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:53.636359 7478 ts_tablet_manager.cc:1929] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:53.636832 7478 log.cc:1199] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/b42fa238590b47d5bc0bd43dd3a313f2
I20260317 21:18:53.637532 7478 ts_tablet_manager.cc:1950] T b42fa238590b47d5bc0bd43dd3a313f2 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:53.645056 7462 tablet_bootstrap.cc:492] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:53.645090 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet 60f3d32e1f7649fa92861f51134cf000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:53.645545 7462 ts_tablet_manager.cc:1403] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.075s user 0.008s sys 0.003s
I20260317 21:18:53.653714 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 50) took 1164 ms. Trace:
I20260317 21:18:53.654243 7478 rpcz_store.cc:276] 0317 21:18:52.489707 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:52.490115 (+ 408us) service_pool.cc:225] Handling call
0317 21:18:53.312093 (+821978us) ts_tablet_manager.cc:1151] Deleting tablet b42fa238590b47d5bc0bd43dd3a313f2
0317 21:18:53.312165 (+ 72us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:53.504743 (+192578us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:53.636108 (+131365us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:53.653626 (+ 17518us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1617,"spinlock_wait_cycles":4352,"tablet-delete.queue_time_us":820638}
I20260317 21:18:53.656176 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet b42fa238590b47d5bc0bd43dd3a313f2 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:18:53.657148 7478 tablet_replica.cc:333] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:53.657838 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_396735bd5352499580e29038e6317fd4 [id=2ed77cec958e4b008989a60fb67a2adc]). Expected version 2 got 1
I20260317 21:18:53.658731 7478 raft_consensus.cc:2243] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:53.659225 7478 raft_consensus.cc:2272] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.661373 7478 ts_tablet_manager.cc:1916] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.666950 7462 raft_consensus.cc:359] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.667877 7462 raft_consensus.cc:385] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.668175 7462 raft_consensus.cc:740] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.673780 7576 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:53.677515 7590 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:53.681305 7467 tablet_bootstrap.cc:492] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:53.681738 7467 ts_tablet_manager.cc:1403] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.306s user 0.018s sys 0.020s
I20260317 21:18:53.686776 7462 consensus_queue.cc:260] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.685992 7467 raft_consensus.cc:359] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.693346 7467 raft_consensus.cc:385] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.693631 7467 raft_consensus.cc:740] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.699369 6973 leader_election.cc:304] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:53.702574 7462 ts_tablet_manager.cc:1434] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
I20260317 21:18:53.709021 7483 ts_tablet_manager.cc:1929] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:53.709386 7483 log.cc:1199] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/bc93b8e8da49471493369deef1a502f3
I20260317 21:18:53.709995 7483 ts_tablet_manager.cc:1950] T bc93b8e8da49471493369deef1a502f3 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:53.714390 7467 consensus_queue.cc:260] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:53.723217 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 88349.5us
I20260317 21:18:53.727384 7483 tablet_replica.cc:333] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:53.728047 7483 raft_consensus.cc:2243] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:53.728818 7483 pending_rounds.cc:70] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Trying to abort 1 pending ops.
I20260317 21:18:53.729064 7483 pending_rounds.cc:77] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412437468114944 op_type: NO_OP noop_request { }
I20260317 21:18:53.729457 7483 raft_consensus.cc:2889] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:53.729779 7483 raft_consensus.cc:2272] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.735540 7580 raft_consensus.cc:493] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:53.736164 7580 raft_consensus.cc:515] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.736881 7483 ts_tablet_manager.cc:1916] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.738380 7580 leader_election.cc:290] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:53.741365 7119 leader_election.cc:343] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: bc93b8e8da49471493369deef1a502f3
I20260317 21:18:53.708215 7477 ts_tablet_manager.cc:1929] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:53.745326 7477 log.cc:1199] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/742b03ea3048466da3c9ea5ed9010b27
I20260317 21:18:53.746198 7477 ts_tablet_manager.cc:1950] T 742b03ea3048466da3c9ea5ed9010b27 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:53.723810 7528 raft_consensus.cc:2749] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:18:53.637619 7459 tablet_bootstrap.cc:492] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:53.749361 7459 ts_tablet_manager.cc:1403] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.610s user 0.013s sys 0.007s
I20260317 21:18:53.750289 7467 ts_tablet_manager.cc:1434] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.068s user 0.006s sys 0.001s
I20260317 21:18:53.755290 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc93b8e8da49471493369deef1a502f3" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:53.756173 7038 raft_consensus.cc:2393] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:18:53.757320 7462 tablet_bootstrap.cc:492] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:53.767640 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 60f3d32e1f7649fa92861f51134cf000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:53.768913 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet bc93b8e8da49471493369deef1a502f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:53.769064 7120 leader_election.cc:304] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:53.770195 7580 raft_consensus.cc:3060] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:53.753269 7459 raft_consensus.cc:359] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.783640 7459 raft_consensus.cc:385] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.783986 7459 raft_consensus.cc:740] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.792809 7467 tablet_bootstrap.cc:492] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:53.793881 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet bc93b8e8da49471493369deef1a502f3 (table default.Table_9f87958031d24bfbb6e275b81361fc4e [id=be78d528072d4de29b4c27d71d6f7034]) successfully deleted
I20260317 21:18:53.794560 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet f41f18434a744b03b6ff53f3d7760e55 from {username='slave'} at 127.0.0.1:39038
W20260317 21:18:53.794744 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet b42fa238590b47d5bc0bd43dd3a313f2 because the tablet was not found. No further retry: Not found: Tablet not found: b42fa238590b47d5bc0bd43dd3a313f2
I20260317 21:18:53.797086 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_544a3d9c42534ca38052e43d62ca136a" } new_table_name: "default.Table_0c987bd2dfe14e3fb5a15492d97199d2" modify_external_catalogs: true
I20260317 21:18:53.797276 7610 tablet.cc:1718] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Skipping requested alter to schema version 2, tablet already version 2
I20260317 21:18:53.799263 7467 tablet_bootstrap.cc:654] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.801966 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954]) successfully deleted
W20260317 21:18:53.802865 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet b42fa238590b47d5bc0bd43dd3a313f2 because the tablet was not found. No further retry: Not found: Tablet not found: b42fa238590b47d5bc0bd43dd3a313f2
W20260317 21:18:53.803862 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 60f3d32e1f7649fa92861f51134cf000 because the tablet was not found. No further retry: Not found: Tablet not found: 60f3d32e1f7649fa92861f51134cf000
I20260317 21:18:53.793442 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e4715bec3faa471cbdcb63c57766180a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:53.805233 7459 consensus_queue.cc:260] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:53.815136 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e4715bec3faa471cbdcb63c57766180a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:53.830803 7459 ts_tablet_manager.cc:1434] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.081s user 0.005s sys 0.000s
I20260317 21:18:53.832072 7610 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20260317 21:18:53.841603 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 46175us
I20260317 21:18:53.844051 7580 raft_consensus.cc:2749] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:53.851991 7459 tablet_bootstrap.cc:492] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:53.875983 7462 tablet_bootstrap.cc:654] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.891731 7467 tablet_bootstrap.cc:492] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:53.892161 7467 ts_tablet_manager.cc:1403] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.004s
W20260317 21:18:53.894495 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e858a01bc4594cc58853ea3239a8dbad has no consensus state
I20260317 21:18:53.894941 7467 raft_consensus.cc:359] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.895565 7467 raft_consensus.cc:385] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.896787 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet f41f18434a744b03b6ff53f3d7760e55
I20260317 21:18:53.898728 6672 catalog_manager.cc:6741] ReplaceTablet: tablet f41f18434a744b03b6ff53f3d7760e55 deleted and replaced by tablet fd7cf19121b5496a849118de4a29db5b
I20260317 21:18:53.900020 7596 raft_consensus.cc:493] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:53.900542 7596 raft_consensus.cc:515] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.902699 7596 leader_election.cc:290] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:18:53.905431 6673 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_396735bd5352499580e29038e6317fd4 [id=2ed77cec958e4b008989a60fb67a2adc]). Expected version 2 got 1
I20260317 21:18:53.906386 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b196f1e1572f4a62b45df7ffc581cd7a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:53.908289 7261 leader_election.cc:343] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:53.909169 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet f41f18434a744b03b6ff53f3d7760e55 with delete_type TABLET_DATA_DELETED (replaced by tablet fd7cf19121b5496a849118de4a29db5b) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:53.909474 7478 ts_tablet_manager.cc:1929] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:53.909809 7478 log.cc:1199] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/f0d55550bf914019b83e83fa4ffcb64b
I20260317 21:18:53.910213 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b196f1e1572f4a62b45df7ffc581cd7a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:53.910450 7478 ts_tablet_manager.cc:1950] T f0d55550bf914019b83e83fa4ffcb64b P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:53.910897 7462 tablet_bootstrap.cc:492] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:53.911345 7462 ts_tablet_manager.cc:1403] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.208s user 0.016s sys 0.000s
I20260317 21:18:53.911379 7483 ts_tablet_manager.cc:1929] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:53.911764 7483 log.cc:1199] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/8deb47bf1eee48a09d8fd6543e409ec3
I20260317 21:18:53.912546 7483 ts_tablet_manager.cc:1950] T 8deb47bf1eee48a09d8fd6543e409ec3 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
W20260317 21:18:53.911712 7264 leader_election.cc:343] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:53.914916 7264 leader_election.cc:304] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:53.915974 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 52) took 1246 ms. Trace:
I20260317 21:18:53.915987 7596 raft_consensus.cc:2749] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:53.916271 7478 rpcz_store.cc:276] 0317 21:18:52.670153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:52.676571 (+ 6418us) service_pool.cc:225] Handling call
0317 21:18:53.656993 (+980422us) ts_tablet_manager.cc:1151] Deleting tablet f0d55550bf914019b83e83fa4ffcb64b
0317 21:18:53.657036 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:53.821264 (+164228us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:53.909246 (+ 87982us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:53.915903 (+ 6657us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":905786}
I20260317 21:18:53.917790 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet f0d55550bf914019b83e83fa4ffcb64b (table default.Table_5bd5abc0a4a64c408b1bae33441413d4 [id=577b3aea931d4e518d588b81917b49c8]) successfully deleted
I20260317 21:18:53.918674 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet b196f1e1572f4a62b45df7ffc581cd7a from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:53.918489 7462 raft_consensus.cc:359] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.919102 7462 raft_consensus.cc:385] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:53.919338 7462 raft_consensus.cc:740] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.919808 7478 tablet_replica.cc:333] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:53.920617 7478 raft_consensus.cc:2243] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:53.922533 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet f41f18434a744b03b6ff53f3d7760e55 with delete_type TABLET_DATA_DELETED (replaced by tablet fd7cf19121b5496a849118de4a29db5b) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:53.926097 7462 consensus_queue.cc:260] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.925953 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet f41f18434a744b03b6ff53f3d7760e55 with delete_type TABLET_DATA_DELETED (replaced by tablet fd7cf19121b5496a849118de4a29db5b) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:53.926954 7478 raft_consensus.cc:2272] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.927333 7477 tablet_replica.cc:333] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:53.928036 7477 raft_consensus.cc:2243] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:53.928520 7477 raft_consensus.cc:2272] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.928900 7478 ts_tablet_manager.cc:1916] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.933591 7477 ts_tablet_manager.cc:1916] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.937975 7462 ts_tablet_manager.cc:1434] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
I20260317 21:18:53.942842 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 55) took 1070 ms. Trace:
I20260317 21:18:53.943277 7483 rpcz_store.cc:276] 0317 21:18:52.872715 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:52.874709 (+ 1994us) service_pool.cc:225] Handling call
0317 21:18:53.727166 (+852457us) ts_tablet_manager.cc:1151] Deleting tablet 8deb47bf1eee48a09d8fd6543e409ec3
0317 21:18:53.727277 (+ 111us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:53.829363 (+102086us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:53.911129 (+ 81766us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:53.942767 (+ 31638us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":851856}
I20260317 21:18:53.945189 7483 tablet_replica.cc:333] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:53.945902 7483 raft_consensus.cc:2243] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:53.946358 7483 raft_consensus.cc:2272] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:53.946949 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 8deb47bf1eee48a09d8fd6543e409ec3 (table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]) successfully deleted
I20260317 21:18:53.948024 7483 ts_tablet_manager.cc:1916] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:53.951263 7462 tablet_bootstrap.cc:492] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:53.895836 7467 raft_consensus.cc:740] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:53.962961 7462 tablet_bootstrap.cc:654] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.962960 7467 consensus_queue.cc:260] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:53.966833 7459 tablet_bootstrap.cc:654] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:53.979907 6669 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4aa5ce0a35044d249a974aee7ccb904b" } modify_external_catalogs: true
I20260317 21:18:53.980754 6669 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4aa5ce0a35044d249a974aee7ccb904b" } modify_external_catalogs: true
I20260317 21:18:53.981068 7467 ts_tablet_manager.cc:1434] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.089s user 0.003s sys 0.006s
I20260317 21:18:53.986805 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d8df008b6b284340ab0c3a41102c096f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:53.988340 7462 tablet_bootstrap.cc:492] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:53.988786 7462 ts_tablet_manager.cc:1403] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.003s
I20260317 21:18:53.989308 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_ae0991c6e7cf4fc1918875b5b48c1062"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.003412 7462 raft_consensus.cc:359] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.004125 7462 raft_consensus.cc:385] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.005016 7467 tablet_bootstrap.cc:492] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:54.004403 7462 raft_consensus.cc:740] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.006999 7462 consensus_queue.cc:260] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.012586 6670 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_396735bd5352499580e29038e6317fd4 [id=2ed77cec958e4b008989a60fb67a2adc] alter complete (version 2)
I20260317 21:18:54.017187 7462 ts_tablet_manager.cc:1434] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.028s user 0.007s sys 0.000s
I20260317 21:18:54.027427 6669 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 655059cfe8d04257957cf9c3cc70cd4c
I20260317 21:18:54.029816 7462 tablet_bootstrap.cc:492] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.033892 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:54.037204 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae0991c6e7cf4fc1918875b5b48c1062 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:54.042428 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b6fb7cb082cf47ce8cf2a98146633af7 has no consensus state
I20260317 21:18:54.046227 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.046948 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet bc93b8e8da49471493369deef1a502f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:54.048352 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d8df008b6b284340ab0c3a41102c096f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.050351 7467 tablet_bootstrap.cc:654] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.057591 7462 tablet_bootstrap.cc:654] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.062538 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 3360925bb5e2415db39a551800ca28d0 with delete_type TABLET_DATA_DELETED (replaced by tablet 60f3d32e1f7649fa92861f51134cf000) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.070490 7462 tablet_bootstrap.cc:492] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:54.070915 7462 ts_tablet_manager.cc:1403] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.008s
I20260317 21:18:54.080972 7462 raft_consensus.cc:359] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.081239 7459 tablet_bootstrap.cc:492] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:54.081640 7462 raft_consensus.cc:385] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.081920 7462 raft_consensus.cc:740] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.082360 7459 ts_tablet_manager.cc:1403] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.231s user 0.008s sys 0.008s
I20260317 21:18:54.083148 7462 consensus_queue.cc:260] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.090497 7467 tablet_bootstrap.cc:492] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:54.090920 7467 ts_tablet_manager.cc:1403] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.086s user 0.010s sys 0.002s
I20260317 21:18:54.095551 7467 raft_consensus.cc:359] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:54.096289 7467 raft_consensus.cc:385] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.096540 7467 raft_consensus.cc:740] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.097955 7459 raft_consensus.cc:359] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.098726 7459 raft_consensus.cc:385] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.103936 7467 consensus_queue.cc:260] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:54.099030 7459 raft_consensus.cc:740] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.106091 7459 consensus_queue.cc:260] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.114949 7462 ts_tablet_manager.cc:1434] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.044s user 0.007s sys 0.002s
I20260317 21:18:54.117491 7483 ts_tablet_manager.cc:1929] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:54.117990 7483 log.cc:1199] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/742b03ea3048466da3c9ea5ed9010b27
I20260317 21:18:54.118788 7483 ts_tablet_manager.cc:1950] T 742b03ea3048466da3c9ea5ed9010b27 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:54.121338 7462 tablet_bootstrap.cc:492] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.133129 7596 raft_consensus.cc:493] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.133916 7596 raft_consensus.cc:515] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:54.136988 7467 ts_tablet_manager.cc:1434] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.046s user 0.011s sys 0.000s
I20260317 21:18:54.138075 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27977b00ae1a4e7ca2eddaa693d11a65" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:54.138938 7183 raft_consensus.cc:2468] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:18:54.140805 7261 leader_election.cc:304] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:54.142155 7549 raft_consensus.cc:2804] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:54.141820 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e938de6ad6044c138f1a4502dac19c9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.142726 7549 raft_consensus.cc:493] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:54.143185 7549 raft_consensus.cc:3060] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:54.142151 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b88a46682c9e48a8a2ed67ad3a8f47f7" } new_table_name: "default.Table_16cf89af28ec48e59cac714020a22b8b" modify_external_catalogs: true
I20260317 21:18:54.138753 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27977b00ae1a4e7ca2eddaa693d11a65" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:54.153057 7596 leader_election.cc:290] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:54.155866 7467 tablet_bootstrap.cc:492] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:54.144280 7459 ts_tablet_manager.cc:1434] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.061s user 0.002s sys 0.004s
W20260317 21:18:54.165122 7264 leader_election.cc:343] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.168467 7459 tablet_bootstrap.cc:492] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:18:54.174777 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e938de6ad6044c138f1a4502dac19c9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.175856 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_02290a1a580945fc935a328e9a9faa11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:54.182112 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_02290a1a580945fc935a328e9a9faa11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.183706 7467 tablet_bootstrap.cc:654] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.187295 7483 tablet_replica.cc:333] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:54.188037 7483 raft_consensus.cc:2243] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:54.189466 7477 ts_tablet_manager.cc:1929] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260317 21:18:54.189888 7477 log.cc:1199] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/f41f18434a744b03b6ff53f3d7760e55
I20260317 21:18:54.190575 7477 ts_tablet_manager.cc:1950] T f41f18434a744b03b6ff53f3d7760e55 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:54.188508 7483 raft_consensus.cc:2272] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.191540 7478 ts_tablet_manager.cc:1929] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:54.191933 7478 log.cc:1199] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/742b03ea3048466da3c9ea5ed9010b27
I20260317 21:18:54.192595 7478 ts_tablet_manager.cc:1950] T 742b03ea3048466da3c9ea5ed9010b27 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:54.192615 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954]) successfully deleted
W20260317 21:18:54.193476 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet c1e672ca556e400ab05763abcd49ad77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1e672ca556e400ab05763abcd49ad77 already in progress: creating tablet
I20260317 21:18:54.192411 7549 raft_consensus.cc:515] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
W20260317 21:18:54.194204 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 60f3d32e1f7649fa92861f51134cf000 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60f3d32e1f7649fa92861f51134cf000 already in progress: creating tablet
I20260317 21:18:54.195161 7483 ts_tablet_manager.cc:1916] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.198369 7549 leader_election.cc:290] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:54.199470 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27977b00ae1a4e7ca2eddaa693d11a65" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:54.200052 7183 raft_consensus.cc:3060] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:54.204023 6662 consensus_peers.cc:597] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 -> Peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Couldn't send request to peer 47e1c983d0224428a9a7b147a0e5e4ef. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20260317 21:18:54.206256 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 3ef4a015f5c14ee6afd045a87eaa6a89 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.207891 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "27977b00ae1a4e7ca2eddaa693d11a65" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
W20260317 21:18:54.210049 7264 leader_election.cc:343] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.223589 7459 tablet_bootstrap.cc:654] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.236899 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 58) took 1036 ms. Trace:
I20260317 21:18:54.237392 7478 rpcz_store.cc:276] 0317 21:18:53.200084 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:53.201902 (+ 1818us) service_pool.cc:225] Handling call
0317 21:18:53.919623 (+717721us) ts_tablet_manager.cc:1151] Deleting tablet 742b03ea3048466da3c9ea5ed9010b27
0317 21:18:53.919684 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:54.114273 (+194589us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:54.191292 (+ 77019us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:54.236806 (+ 45514us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":716730}
I20260317 21:18:54.240106 7477 tablet_replica.cc:333] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:54.240087 7183 raft_consensus.cc:2468] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:18:54.240979 7477 raft_consensus.cc:2243] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:54.241470 7477 raft_consensus.cc:2272] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.242239 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]) successfully deleted
I20260317 21:18:54.242601 7261 leader_election.cc:304] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.244541 7549 raft_consensus.cc:2804] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:54.245227 7549 raft_consensus.cc:697] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:18:54.246651 7549 consensus_queue.cc:237] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:54.250262 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1831269296644fc18ad3b11bdde06abd" } new_table_name: "default.Table_4d068d09b4944465bfdf018355207dc3" modify_external_catalogs: true
I20260317 21:18:54.253183 7580 raft_consensus.cc:493] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.253619 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 742b03ea3048466da3c9ea5ed9010b27 (table default.Table_eec7ee55d8bf4e9baa04680fd31ac46c [id=bfdb4aafb98244168a6123384b829954]) successfully deleted
I20260317 21:18:54.254050 7580 raft_consensus.cc:515] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.258061 7478 tablet_replica.cc:333] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:54.258985 7478 raft_consensus.cc:2243] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:54.259512 7478 raft_consensus.cc:2272] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.261929 7580 leader_election.cc:290] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.192548 7462 tablet_bootstrap.cc:654] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:54.267338 7119 leader_election.cc:343] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:54.269942 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4493a733fd0a496885b7745fcfc6cf2e" } modify_external_catalogs: true
I20260317 21:18:54.272799 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 291e2bee709243f09b338d99edbe0cb1
I20260317 21:18:54.276007 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 589aadc7897b422e90d3f7caed76ab59 (DEFAULT_TABLE table=default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:54.276714 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 935faf7f9129440a91635bfa96191cd6 consensus state has no leader
I20260317 21:18:54.277243 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 589aadc7897b422e90d3f7caed76ab59 (DEFAULT_TABLE table=default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.278134 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589aadc7897b422e90d3f7caed76ab59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.283125 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet 291e2bee709243f09b338d99edbe0cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:53 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:54.284412 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 291e2bee709243f09b338d99edbe0cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:53 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:54.289511 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_2e2110a652674960aa102cd23392bd89"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.271552 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4493a733fd0a496885b7745fcfc6cf2e" } modify_external_catalogs: true
W20260317 21:18:54.292900 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2e2110a652674960aa102cd23392bd89 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:54.295738 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b196f1e1572f4a62b45df7ffc581cd7a consensus state has no leader
I20260317 21:18:54.310678 7308 tablet_service.cc:1511] Processing CreateTablet for tablet f7ec20308a9244a49992e378fa7b4f6c (DEFAULT_TABLE table=default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.312448 7154 tablet_service.cc:1511] Processing CreateTablet for tablet f7ec20308a9244a49992e378fa7b4f6c (DEFAULT_TABLE table=default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.314503 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ec20308a9244a49992e378fa7b4f6c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:54.315405 7119 consensus_peers.cc:597] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:54.319067 7528 raft_consensus.cc:493] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.319891 7528 raft_consensus.cc:515] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:54.324553 7120 consensus_peers.cc:597] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:54.327785 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f4e1fbb0d24813b0758cae9477f006" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:54.328344 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 6f980bd133da4b04b101e8df99ccca6f (DEFAULT_TABLE table=default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.330154 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f980bd133da4b04b101e8df99ccca6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.333247 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 8deb47bf1eee48a09d8fd6543e409ec3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:50 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:54.334515 7120 leader_election.cc:343] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.335178 7120 leader_election.cc:304] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.337203 7477 ts_tablet_manager.cc:1916] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.337823 7580 raft_consensus.cc:2749] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:54.342176 6660 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet b196f1e1572f4a62b45df7ffc581cd7a (table default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b196f1e1572f4a62b45df7ffc581cd7a consensus state has no leader
I20260317 21:18:54.350692 7008 tablet_service.cc:1511] Processing CreateTablet for tablet f7ec20308a9244a49992e378fa7b4f6c (DEFAULT_TABLE table=default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:54.351673 6971 leader_election.cc:343] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:54.355350 6668 master_service.cc:954] ReplaceTablet: received request to replace tablet 163cc8c04ca94a269c7ae2efaa162525 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:54.356355 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 589aadc7897b422e90d3f7caed76ab59 (DEFAULT_TABLE table=default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.353098 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet 291e2bee709243f09b338d99edbe0cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:53 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.352541 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ec20308a9244a49992e378fa7b4f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.357213 7528 leader_election.cc:290] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.362098 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589aadc7897b422e90d3f7caed76ab59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.362828 7462 tablet_bootstrap.cc:492] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:54.364094 7462 ts_tablet_manager.cc:1403] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.243s user 0.010s sys 0.004s
W20260317 21:18:54.365373 6973 leader_election.cc:343] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:54.366091 6973 leader_election.cc:304] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:54.368955 7528 raft_consensus.cc:2749] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:54.377769 7459 tablet_bootstrap.cc:492] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:54.378269 7459 ts_tablet_manager.cc:1403] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.210s user 0.018s sys 0.011s
I20260317 21:18:54.382397 7459 raft_consensus.cc:359] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.383069 7459 raft_consensus.cc:385] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.383328 7459 raft_consensus.cc:740] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.384117 7462 raft_consensus.cc:359] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.385542 7462 raft_consensus.cc:385] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.385851 7462 raft_consensus.cc:740] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.386646 7462 consensus_queue.cc:260] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.267318 7478 ts_tablet_manager.cc:1916] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.397238 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 589aadc7897b422e90d3f7caed76ab59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.399506 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f7ec20308a9244a49992e378fa7b4f6c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.412281 7462 ts_tablet_manager.cc:1434] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.047s user 0.007s sys 0.001s
I20260317 21:18:54.417194 7483 ts_tablet_manager.cc:1929] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260317 21:18:54.417744 7483 log.cc:1199] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/f41f18434a744b03b6ff53f3d7760e55
I20260317 21:18:54.418792 7483 ts_tablet_manager.cc:1950] T f41f18434a744b03b6ff53f3d7760e55 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:54.420734 7462 tablet_bootstrap.cc:492] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.430500 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 6f980bd133da4b04b101e8df99ccca6f (DEFAULT_TABLE table=default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.431586 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 6f980bd133da4b04b101e8df99ccca6f (DEFAULT_TABLE table=default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.433172 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f980bd133da4b04b101e8df99ccca6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.433902 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f980bd133da4b04b101e8df99ccca6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.437906 7467 tablet_bootstrap.cc:492] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:54.438360 7467 ts_tablet_manager.cc:1403] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.283s user 0.006s sys 0.010s
I20260317 21:18:54.441140 7467 raft_consensus.cc:359] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.441833 7483 tablet_replica.cc:333] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:54.441856 7467 raft_consensus.cc:385] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.442109 7467 raft_consensus.cc:740] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.442111 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]) successfully deleted
I20260317 21:18:54.442669 7467 consensus_queue.cc:260] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.444711 7007 tablet_service.cc:1511] Processing CreateTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 (DEFAULT_TABLE table=default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.445936 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fb7cb082cf47ce8cf2a98146633af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.448846 7483 raft_consensus.cc:2243] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:54.452814 7483 raft_consensus.cc:2272] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.454820 7483 ts_tablet_manager.cc:1916] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.473140 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet b196f1e1572f4a62b45df7ffc581cd7a
I20260317 21:18:54.474924 6672 catalog_manager.cc:6741] ReplaceTablet: tablet b196f1e1572f4a62b45df7ffc581cd7a deleted and replaced by tablet a978fa28a31b4d589031c77526b62b8e
I20260317 21:18:54.476867 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_212ed8405de344519eaecc32555fcbb5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.477667 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a with delete_type TABLET_DATA_DELETED (replaced by tablet a978fa28a31b4d589031c77526b62b8e) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:54.478809 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_212ed8405de344519eaecc32555fcbb5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.485299 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a with delete_type TABLET_DATA_DELETED (replaced by tablet a978fa28a31b4d589031c77526b62b8e) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.488360 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0c38032670874e6f85493a54b207749e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:54.490350 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0c38032670874e6f85493a54b207749e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.383939 7459 consensus_queue.cc:260] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.496189 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f701f4a89ae64b4db661f19343773e30"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.497018 7459 ts_tablet_manager.cc:1434] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.118s user 0.009s sys 0.000s
W20260317 21:18:54.498142 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f701f4a89ae64b4db661f19343773e30 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.503886 7467 ts_tablet_manager.cc:1434] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.065s user 0.006s sys 0.001s
I20260317 21:18:54.508711 7459 tablet_bootstrap.cc:492] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:54.509186 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a with delete_type TABLET_DATA_DELETED (replaced by tablet a978fa28a31b4d589031c77526b62b8e) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:54.511806 6670 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_5bd5abc0a4a64c408b1bae33441413d4" } new_table_name: "default.Table_34196bd319d74982a3e5f88cb0957c14" modify_external_catalogs: true
I20260317 21:18:54.513605 6673 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_afb9c6074ee54f86a4d029cf8cdbb825" } modify_external_catalogs: true
I20260317 21:18:54.513552 7154 tablet_service.cc:1511] Processing CreateTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 (DEFAULT_TABLE table=default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.515012 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6dc2ff9e049443948de3753522e042f8" } new_table_name: "default.Table_b8fd10368ce242839da23252952d7600" modify_external_catalogs: true
I20260317 21:18:54.519366 7467 tablet_bootstrap.cc:492] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:54.519826 7304 tablet_service.cc:1511] Processing CreateTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 (DEFAULT_TABLE table=default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.521001 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fb7cb082cf47ce8cf2a98146633af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.525177 7462 tablet_bootstrap.cc:654] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.530804 6673 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_afb9c6074ee54f86a4d029cf8cdbb825" } modify_external_catalogs: true
I20260317 21:18:54.539426 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet c6d5385171364a078d968bb0a489383a
I20260317 21:18:54.543259 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet c6d5385171364a078d968bb0a489383a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:54.546593 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet c6d5385171364a078d968bb0a489383a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:54.553136 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b6fb7cb082cf47ce8cf2a98146633af7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.559069 7549 raft_consensus.cc:493] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.559708 7549 raft_consensus.cc:515] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.563844 7549 leader_election.cc:290] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:54.583256 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 82f8dd260a134810a50465e4a77848ca (DEFAULT_TABLE table=default.Table_8cba5340b2044350864c86b46e1a9488 [id=bcbc5e44c4ce46d2b795f6997fa466ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.584425 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f8dd260a134810a50465e4a77848ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.590222 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 82f8dd260a134810a50465e4a77848ca (DEFAULT_TABLE table=default.Table_8cba5340b2044350864c86b46e1a9488 [id=bcbc5e44c4ce46d2b795f6997fa466ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.592681 7467 tablet_bootstrap.cc:654] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.593248 6668 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 163cc8c04ca94a269c7ae2efaa162525
I20260317 21:18:54.593096 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:54.594676 6668 catalog_manager.cc:6741] ReplaceTablet: tablet 163cc8c04ca94a269c7ae2efaa162525 deleted and replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1
W20260317 21:18:54.594537 7261 leader_election.cc:343] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.599409 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:54.601498 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:54.602087 7462 tablet_bootstrap.cc:492] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:54.609122 7462 ts_tablet_manager.cc:1403] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.189s user 0.017s sys 0.004s
I20260317 21:18:54.610983 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 82f8dd260a134810a50465e4a77848ca (DEFAULT_TABLE table=default.Table_8cba5340b2044350864c86b46e1a9488 [id=bcbc5e44c4ce46d2b795f6997fa466ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.617746 7467 tablet_bootstrap.cc:492] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:54.618192 7467 ts_tablet_manager.cc:1403] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.099s user 0.010s sys 0.005s
I20260317 21:18:54.621068 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:54.622174 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet c6d5385171364a078d968bb0a489383a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.618916 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.623719 7462 raft_consensus.cc:359] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.624377 7462 raft_consensus.cc:385] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.630316 7467 raft_consensus.cc:359] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:54.630817 7264 leader_election.cc:343] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.631011 7467 raft_consensus.cc:385] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.631280 7467 raft_consensus.cc:740] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.631253 7264 leader_election.cc:304] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.624684 7462 raft_consensus.cc:740] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.632005 7549 raft_consensus.cc:2749] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:54.632391 7462 consensus_queue.cc:260] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.634578 7462 ts_tablet_manager.cc:1434] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.025s user 0.007s sys 0.000s
I20260317 21:18:54.635461 7462 tablet_bootstrap.cc:492] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.637539 7478 ts_tablet_manager.cc:1929] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:54.638131 7467 consensus_queue.cc:260] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.638684 7478 log.cc:1199] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/8deb47bf1eee48a09d8fd6543e409ec3
I20260317 21:18:54.639443 7478 ts_tablet_manager.cc:1950] T 8deb47bf1eee48a09d8fd6543e409ec3 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:54.640211 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f8dd260a134810a50465e4a77848ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.652453 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 2562cc2a3bb2430981c88475f9e1daa1 (DEFAULT_TABLE table=default.Table_d50638a040344b81ad504d963feb3a4f [id=6d460b03e35a454da7657c608a0f87cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.657506 7467 ts_tablet_manager.cc:1434] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.039s user 0.008s sys 0.003s
I20260317 21:18:54.664920 7467 tablet_bootstrap.cc:492] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:54.671849 7467 tablet_bootstrap.cc:654] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.675976 7462 tablet_bootstrap.cc:654] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.679100 7459 tablet_bootstrap.cc:654] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.645434 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 2562cc2a3bb2430981c88475f9e1daa1 (DEFAULT_TABLE table=default.Table_d50638a040344b81ad504d963feb3a4f [id=6d460b03e35a454da7657c608a0f87cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.680706 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2562cc2a3bb2430981c88475f9e1daa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.686502 7549 raft_consensus.cc:493] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.687119 7549 raft_consensus.cc:515] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.604990 6675 catalog_manager.cc:5671] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67). New cstate: current_term: 1 leader_uuid: "2f539d1a4a584b8ca83928be49369bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:54.691639 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:54.693315 7261 leader_election.cc:343] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.693811 7549 leader_election.cc:290] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:54.694159 7462 tablet_bootstrap.cc:492] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:54.694566 7462 ts_tablet_manager.cc:1403] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.059s user 0.014s sys 0.003s
I20260317 21:18:54.697513 7462 raft_consensus.cc:359] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.698318 7462 raft_consensus.cc:385] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.698587 7462 raft_consensus.cc:740] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.700934 7462 consensus_queue.cc:260] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.704843 7462 ts_tablet_manager.cc:1434] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20260317 21:18:54.705725 7462 tablet_bootstrap.cc:492] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.711333 7462 tablet_bootstrap.cc:654] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.715544 7158 tablet_service.cc:1511] Processing CreateTablet for tablet c6d5385171364a078d968bb0a489383a (DEFAULT_TABLE table=default.Table_4493a733fd0a496885b7745fcfc6cf2e [id=e8d1164df252476a82baf64ec8c6c9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.716821 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d5385171364a078d968bb0a489383a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.721771 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 2562cc2a3bb2430981c88475f9e1daa1 (DEFAULT_TABLE table=default.Table_d50638a040344b81ad504d963feb3a4f [id=6d460b03e35a454da7657c608a0f87cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.724881 7305 tablet_service.cc:1511] Processing CreateTablet for tablet c6d5385171364a078d968bb0a489383a (DEFAULT_TABLE table=default.Table_4493a733fd0a496885b7745fcfc6cf2e [id=e8d1164df252476a82baf64ec8c6c9cf]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:54.729975 6670 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4522181cc434203ac14c9b07587fec8 consensus state has no leader
I20260317 21:18:54.736987 7580 raft_consensus.cc:493] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.737547 7580 raft_consensus.cc:515] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.738411 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 59) took 1535 ms. Trace:
I20260317 21:18:54.738734 7478 rpcz_store.cc:276] 0317 21:18:53.203592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:53.204399 (+ 807us) service_pool.cc:225] Handling call
0317 21:18:54.255244 (+1050845us) ts_tablet_manager.cc:1151] Deleting tablet 8deb47bf1eee48a09d8fd6543e409ec3
0317 21:18:54.255344 (+ 100us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:54.537040 (+281696us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:54.637295 (+100255us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:54.738354 (+101059us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1050134}
I20260317 21:18:54.739389 7580 leader_election.cc:290] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.742533 7467 tablet_bootstrap.cc:492] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:54.742985 7467 ts_tablet_manager.cc:1403] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.078s user 0.015s sys 0.010s
I20260317 21:18:54.743829 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 8deb47bf1eee48a09d8fd6543e409ec3 (table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]) successfully deleted
I20260317 21:18:54.745697 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be2c469c792047d88fcf2ff9d53b7832" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:54.746528 7329 raft_consensus.cc:2468] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:54.691268 7477 ts_tablet_manager.cc:1929] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:54.693049 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 82f8dd260a134810a50465e4a77848ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.749358 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2562cc2a3bb2430981c88475f9e1daa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.695465 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:54.712271 7017 tablet_service.cc:1511] Processing CreateTablet for tablet c6d5385171364a078d968bb0a489383a (DEFAULT_TABLE table=default.Table_4493a733fd0a496885b7745fcfc6cf2e [id=e8d1164df252476a82baf64ec8c6c9cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.716079 7009 tablet_service.cc:1511] Processing CreateTablet for tablet e858a01bc4594cc58853ea3239a8dbad (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.750241 7467 raft_consensus.cc:359] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.750953 7467 raft_consensus.cc:385] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.751226 7467 raft_consensus.cc:740] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.751701 7477 log.cc:1199] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/8deb47bf1eee48a09d8fd6543e409ec3
I20260317 21:18:54.752148 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d5385171364a078d968bb0a489383a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.751784 7467 consensus_queue.cc:260] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.752800 7119 leader_election.cc:304] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:54.753077 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e858a01bc4594cc58853ea3239a8dbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.753522 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 60) took 1464 ms. Trace:
I20260317 21:18:54.753836 7478 rpcz_store.cc:276] 0317 21:18:53.289818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:53.290025 (+ 207us) service_pool.cc:225] Handling call
0317 21:18:54.739765 (+1449740us) ts_tablet_manager.cc:1151] Deleting tablet c1e672ca556e400ab05763abcd49ad77
0317 21:18:54.739803 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:54.753462 (+ 13659us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1448771}
I20260317 21:18:54.759097 7580 raft_consensus.cc:2804] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:54.759498 7580 raft_consensus.cc:493] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:54.759886 7580 raft_consensus.cc:3060] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:54.761065 7477 ts_tablet_manager.cc:1950] T 8deb47bf1eee48a09d8fd6543e409ec3 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:54.762506 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 8deb47bf1eee48a09d8fd6543e409ec3 (table default.Table_5fce106f6b6a4e00b9af07d5e5fc930f [id=e735bfe151b64988aae7b0967654075e]) successfully deleted
I20260317 21:18:54.762714 7477 tablet_replica.cc:333] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:54.763742 7477 raft_consensus.cc:2243] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:54.766196 7467 ts_tablet_manager.cc:1434] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.023s user 0.005s sys 0.001s
I20260317 21:18:54.767442 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 64) took 1168 ms. Trace:
I20260317 21:18:54.767593 7528 raft_consensus.cc:493] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.767851 7478 rpcz_store.cc:276] 0317 21:18:53.599613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:53.644860 (+ 45247us) service_pool.cc:225] Handling call
0317 21:18:54.754764 (+1109904us) ts_tablet_manager.cc:1151] Deleting tablet 60f3d32e1f7649fa92861f51134cf000
0317 21:18:54.766975 (+ 12211us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:54.767361 (+ 386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109034}
I20260317 21:18:54.768298 7528 raft_consensus.cc:515] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.768779 7478 tablet_replica.cc:333] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:54.769506 7478 raft_consensus.cc:2243] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:54.770097 7478 raft_consensus.cc:2272] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.770301 7528 leader_election.cc:290] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.770675 7462 tablet_bootstrap.cc:492] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:54.771085 7462 ts_tablet_manager.cc:1403] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.066s user 0.009s sys 0.003s
I20260317 21:18:54.771759 7478 ts_tablet_manager.cc:1916] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.773012 7467 tablet_bootstrap.cc:492] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:18:54.774780 6971 leader_election.cc:343] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 3360925bb5e2415db39a551800ca28d0
W20260317 21:18:54.775499 6973 leader_election.cc:343] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3360925bb5e2415db39a551800ca28d0
I20260317 21:18:54.775894 6973 leader_election.cc:304] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:54.777303 7528 raft_consensus.cc:2749] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:54.778060 7462 raft_consensus.cc:359] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.778755 7462 raft_consensus.cc:385] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.779013 7462 raft_consensus.cc:740] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.779623 7462 consensus_queue.cc:260] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:54.787256 7264 leader_election.cc:343] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.790377 7549 raft_consensus.cc:493] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.791082 7549 raft_consensus.cc:515] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.764389 7477 raft_consensus.cc:2272] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.794101 7477 ts_tablet_manager.cc:1916] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.799124 7467 tablet_bootstrap.cc:654] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:54.806246 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet c1e672ca556e400ab05763abcd49ad77 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c1e672ca556e400ab05763abcd49ad77 already in progress: creating tablet
I20260317 21:18:54.812160 7630 raft_consensus.cc:493] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.812927 7630 raft_consensus.cc:515] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:54.819092 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 60f3d32e1f7649fa92861f51134cf000 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60f3d32e1f7649fa92861f51134cf000 already in progress: creating tablet
I20260317 21:18:54.819970 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2562cc2a3bb2430981c88475f9e1daa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.820288 7462 ts_tablet_manager.cc:1434] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.049s user 0.006s sys 0.001s
I20260317 21:18:54.820379 7549 leader_election.cc:290] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:54.821271 7462 tablet_bootstrap.cc:492] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.822266 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6d5385171364a078d968bb0a489383a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.787823 7264 leader_election.cc:304] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.824766 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b196f1e1572f4a62b45df7ffc581cd7a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:54.826851 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56495c7cf3964743b96ab885afdaa152" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:54.828778 7120 leader_election.cc:343] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:54.829020 7261 leader_election.cc:343] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.719010 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 33f4e1fbb0d24813b0758cae9477f006 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.744000 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be2c469c792047d88fcf2ff9d53b7832" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:54.833010 7120 leader_election.cc:343] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.833923 7580 raft_consensus.cc:515] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.836128 7580 leader_election.cc:290] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.839351 7630 leader_election.cc:290] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.841287 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be2c469c792047d88fcf2ff9d53b7832" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:54.839823 7462 tablet_bootstrap.cc:654] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.853484 7549 raft_consensus.cc:2749] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:54.854596 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 67357.5us
I20260317 21:18:54.857259 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56495c7cf3964743b96ab885afdaa152" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:54.859781 7154 tablet_service.cc:1511] Processing CreateTablet for tablet e858a01bc4594cc58853ea3239a8dbad (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.860530 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b196f1e1572f4a62b45df7ffc581cd7a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:54.861317 7329 raft_consensus.cc:2468] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:54.863003 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "be2c469c792047d88fcf2ff9d53b7832" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:54.863772 7328 raft_consensus.cc:3060] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:54.868175 7120 leader_election.cc:343] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.869074 7630 raft_consensus.cc:493] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:54.869614 7630 raft_consensus.cc:515] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.873543 6673 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 53dd9f8a1ccc47f884bfba99781aed16
W20260317 21:18:54.875169 7264 leader_election.cc:343] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.875131 7119 leader_election.cc:304] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.875607 7264 leader_election.cc:304] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.876073 7580 raft_consensus.cc:2804] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:54.876448 7580 raft_consensus.cc:493] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:54.876888 7580 raft_consensus.cc:3060] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:54.878463 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "291e2bee709243f09b338d99edbe0cb1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:54.881525 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e1094068903840a8ab5d07c7fb0635bf consensus state has no leader
W20260317 21:18:54.881721 7120 leader_election.cc:343] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.882262 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_2e2110a652674960aa102cd23392bd89" } modify_external_catalogs: true
I20260317 21:18:54.884289 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_2e2110a652674960aa102cd23392bd89" } modify_external_catalogs: true
I20260317 21:18:54.888939 7630 leader_election.cc:290] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:54.891417 7462 tablet_bootstrap.cc:492] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:54.891952 7462 ts_tablet_manager.cc:1403] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.071s user 0.006s sys 0.005s
I20260317 21:18:54.893880 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:54.894456 7549 raft_consensus.cc:2749] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:54.897238 7304 tablet_service.cc:1511] Processing CreateTablet for tablet e858a01bc4594cc58853ea3239a8dbad (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.899673 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "291e2bee709243f09b338d99edbe0cb1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:18:54.901964 7119 leader_election.cc:343] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:54.902417 7119 leader_election.cc:304] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.903792 7630 raft_consensus.cc:2749] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:54.909541 7462 raft_consensus.cc:359] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.911998 7462 raft_consensus.cc:385] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:54.912802 7462 raft_consensus.cc:740] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:54.914149 7462 consensus_queue.cc:260] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.890311 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f709509873824f4d86f7018329a3e917"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.919389 7462 ts_tablet_manager.cc:1434] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.027s user 0.004s sys 0.004s
W20260317 21:18:54.922297 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f709509873824f4d86f7018329a3e917 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.924114 7462 tablet_bootstrap.cc:492] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:54.934783 7478 ts_tablet_manager.cc:1929] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:54.935326 7478 log.cc:1199] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/bc93b8e8da49471493369deef1a502f3
I20260317 21:18:54.935987 7478 ts_tablet_manager.cc:1950] T bc93b8e8da49471493369deef1a502f3 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:54.936551 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e858a01bc4594cc58853ea3239a8dbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.936545 7462 tablet_bootstrap.cc:654] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:54.937644 7328 raft_consensus.cc:2468] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:54.939344 7580 raft_consensus.cc:515] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.941020 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:54.942456 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_8730523559834c72ab08f2cec8b6230a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:54.943256 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e858a01bc4594cc58853ea3239a8dbad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.943483 7483 ts_tablet_manager.cc:1929] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:54.943951 7483 log.cc:1199] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/291e2bee709243f09b338d99edbe0cb1
I20260317 21:18:54.944684 7483 ts_tablet_manager.cc:1950] T 291e2bee709243f09b338d99edbe0cb1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
W20260317 21:18:54.944525 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8730523559834c72ab08f2cec8b6230a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:54.952003 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 935faf7f9129440a91635bfa96191cd6 (DEFAULT_TABLE table=default.Table_16cf89af28ec48e59cac714020a22b8b [id=1e766b4608a743bb9716dbb5d782b3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.953811 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 935faf7f9129440a91635bfa96191cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.954447 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 65) took 1206 ms. Trace:
I20260317 21:18:54.954872 7478 rpcz_store.cc:276] 0317 21:18:53.748154 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:53.768762 (+ 20608us) service_pool.cc:225] Handling call
0317 21:18:54.768598 (+999836us) ts_tablet_manager.cc:1151] Deleting tablet bc93b8e8da49471493369deef1a502f3
0317 21:18:54.768676 (+ 78us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:54.836902 (+ 68226us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:54.934434 (+ 97532us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:54.954384 (+ 19950us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":999230}
I20260317 21:18:54.956313 7478 tablet_replica.cc:333] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:54.956411 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:54.957137 7478 raft_consensus.cc:2243] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:54.957834 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet bc93b8e8da49471493369deef1a502f3 (table default.Table_9f87958031d24bfbb6e275b81361fc4e [id=be78d528072d4de29b4c27d71d6f7034]) successfully deleted
I20260317 21:18:54.960322 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 935faf7f9129440a91635bfa96191cd6 (DEFAULT_TABLE table=default.Table_16cf89af28ec48e59cac714020a22b8b [id=1e766b4608a743bb9716dbb5d782b3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.960886 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b196f1e1572f4a62b45df7ffc581cd7a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:54.961498 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 935faf7f9129440a91635bfa96191cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:54.962735 7580 leader_election.cc:290] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:54.963620 7120 leader_election.cc:343] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:54.965430 7119 leader_election.cc:304] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:54.971815 7580 raft_consensus.cc:2804] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:54.974056 7580 raft_consensus.cc:697] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:54.977985 7478 raft_consensus.cc:2272] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:54.978986 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b196f1e1572f4a62b45df7ffc581cd7a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:54.979628 7328 raft_consensus.cc:3060] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:54.981262 7478 ts_tablet_manager.cc:1916] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:54.977665 7580 consensus_queue.cc:237] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:54.995815 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 935faf7f9129440a91635bfa96191cd6 (DEFAULT_TABLE table=default.Table_16cf89af28ec48e59cac714020a22b8b [id=1e766b4608a743bb9716dbb5d782b3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:54.997066 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 935faf7f9129440a91635bfa96191cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.006912 7467 tablet_bootstrap.cc:492] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:55.007768 7467 ts_tablet_manager.cc:1403] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.235s user 0.020s sys 0.035s
I20260317 21:18:55.021293 7154 tablet_service.cc:1511] Processing CreateTablet for tablet d2f7c1058f8245428f38fe3245621b75 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.022130 6673 master_service.cc:954] ReplaceTablet: received request to replace tablet 27977b00ae1a4e7ca2eddaa693d11a65 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:55.023597 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet b6fb7cb082cf47ce8cf2a98146633af7 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:55.024744 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_34196bd319d74982a3e5f88cb0957c14" } new_table_name: "default.Table_24103dddfd444503aa58762e5d598637" modify_external_catalogs: true
I20260317 21:18:55.025405 7304 tablet_service.cc:1511] Processing CreateTablet for tablet d2f7c1058f8245428f38fe3245621b75 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.026751 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2f7c1058f8245428f38fe3245621b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.037181 7477 ts_tablet_manager.cc:1929] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.037555 7477 log.cc:1199] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/bc93b8e8da49471493369deef1a502f3
I20260317 21:18:55.038206 7477 ts_tablet_manager.cc:1950] T bc93b8e8da49471493369deef1a502f3 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:55.043262 7483 tablet_replica.cc:333] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:55.055812 7328 raft_consensus.cc:2468] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:55.056268 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:55.057340 7119 leader_election.cc:304] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.058475 7580 raft_consensus.cc:2804] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:55.059327 7580 raft_consensus.cc:697] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:55.060761 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 56495c7cf3964743b96ab885afdaa152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.061544 7580 consensus_queue.cc:237] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.063650 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 291e2bee709243f09b338d99edbe0cb1 (table default.Table_2fab17210bfa496ba34bb9580bee9d79 [id=040d2e5e61ef41e4ac8953a3d63a226c]) successfully deleted
W20260317 21:18:55.064301 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 8deb47bf1eee48a09d8fd6543e409ec3 because the tablet was not found. No further retry: Not found: Tablet not found: 8deb47bf1eee48a09d8fd6543e409ec3
I20260317 21:18:55.064329 7483 raft_consensus.cc:2243] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:55.064916 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.068785 7483 raft_consensus.cc:2272] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.088909 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2f7c1058f8245428f38fe3245621b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.092309 7159 tablet_service.cc:1511] Processing CreateTablet for tablet e4522181cc434203ac14c9b07587fec8 (DEFAULT_TABLE table=default.Table_34196bd319d74982a3e5f88cb0957c14 [id=577b3aea931d4e518d588b81917b49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.093508 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4522181cc434203ac14c9b07587fec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.055809 7462 tablet_bootstrap.cc:492] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.101111 7462 ts_tablet_manager.cc:1403] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.177s user 0.025s sys 0.003s
I20260317 21:18:55.102573 7467 raft_consensus.cc:359] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.103453 7467 raft_consensus.cc:385] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.103717 7467 raft_consensus.cc:740] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.103849 7462 raft_consensus.cc:359] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.104627 7462 raft_consensus.cc:385] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.104981 7462 raft_consensus.cc:740] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.105773 7462 consensus_queue.cc:260] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.107429 7013 tablet_service.cc:1511] Processing CreateTablet for tablet d2f7c1058f8245428f38fe3245621b75 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.108623 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2f7c1058f8245428f38fe3245621b75. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.112067 7012 tablet_service.cc:1511] Processing CreateTablet for tablet dc929bae186841d2aba674659e457301 (DEFAULT_TABLE table=default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.113469 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc929bae186841d2aba674659e457301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.114387 7010 tablet_service.cc:1511] Processing CreateTablet for tablet e4522181cc434203ac14c9b07587fec8 (DEFAULT_TABLE table=default.Table_34196bd319d74982a3e5f88cb0957c14 [id=577b3aea931d4e518d588b81917b49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.117348 7462 ts_tablet_manager.cc:1434] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.016s user 0.006s sys 0.000s
I20260317 21:18:55.124444 7459 tablet_bootstrap.cc:492] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:55.125083 7459 ts_tablet_manager.cc:1403] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.617s user 0.012s sys 0.000s
I20260317 21:18:55.126758 7462 tablet_bootstrap.cc:492] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.129524 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4522181cc434203ac14c9b07587fec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.131918 7462 tablet_bootstrap.cc:654] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.136817 7459 raft_consensus.cc:359] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:55.137619 7459 raft_consensus.cc:385] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.137966 7459 raft_consensus.cc:740] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.141314 7462 tablet_bootstrap.cc:492] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.141729 7462 ts_tablet_manager.cc:1403] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.015s user 0.005s sys 0.003s
I20260317 21:18:55.146417 7462 raft_consensus.cc:359] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.147079 7462 raft_consensus.cc:385] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.147327 7462 raft_consensus.cc:740] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.151655 7462 consensus_queue.cc:260] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.156324 7462 ts_tablet_manager.cc:1434] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.014s user 0.003s sys 0.004s
I20260317 21:18:55.157222 7462 tablet_bootstrap.cc:492] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.159996 7459 consensus_queue.cc:260] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:55.162786 7462 tablet_bootstrap.cc:654] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.170077 7459 ts_tablet_manager.cc:1434] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260317 21:18:55.147115 7477 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 62) took 1102 ms. Trace:
I20260317 21:18:55.170537 7477 rpcz_store.cc:276] 0317 21:18:54.044765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.046788 (+ 2023us) service_pool.cc:225] Handling call
0317 21:18:54.762557 (+715769us) ts_tablet_manager.cc:1151] Deleting tablet bc93b8e8da49471493369deef1a502f3
0317 21:18:54.762595 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:54.945324 (+182729us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.036947 (+ 91623us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.147040 (+110093us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":37248,"tablet-delete.queue_time_us":715194}
I20260317 21:18:55.172621 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet bc93b8e8da49471493369deef1a502f3 (table default.Table_9f87958031d24bfbb6e275b81361fc4e [id=be78d528072d4de29b4c27d71d6f7034]) successfully deleted
I20260317 21:18:55.179898 6670 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet b938dfc2d1ce40c88f0d3dfb8ea3c1e1
I20260317 21:18:55.180588 7528 raft_consensus.cc:493] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.181169 7528 raft_consensus.cc:515] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.183636 7528 leader_election.cc:290] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:55.185395 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f4e1fbb0d24813b0758cae9477f006" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:55.186204 7183 raft_consensus.cc:2468] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:55.187830 6971 leader_election.cc:304] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:55.188475 7549 raft_consensus.cc:493] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.188920 7528 raft_consensus.cc:2804] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:55.189313 7528 raft_consensus.cc:493] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:55.189033 7549 raft_consensus.cc:515] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.189684 7528 raft_consensus.cc:3060] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:55.191152 7549 leader_election.cc:290] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:55.193188 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:55.195899 7261 leader_election.cc:343] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:55.197794 6973 leader_election.cc:343] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:55.201741 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:55.203678 7483 ts_tablet_manager.cc:1916] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.204291 7477 tablet_replica.cc:333] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:55.205323 7477 raft_consensus.cc:2243] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
W20260317 21:18:55.203914 7264 leader_election.cc:343] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.205868 7264 leader_election.cc:304] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.206710 7549 raft_consensus.cc:2749] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:55.209594 7467 consensus_queue.cc:260] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:55.210716 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 106116us
I20260317 21:18:55.217816 6670 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet d59deea79cfb4a83831bb0227797c690 (table default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]). Expected version 1 got 0
I20260317 21:18:55.218885 6670 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 06877bd7155842b9a5ae4a23ea68ada0 (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 2 got 0
I20260317 21:18:55.224957 7462 tablet_bootstrap.cc:492] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.224951 7477 raft_consensus.cc:2272] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.225471 7462 ts_tablet_manager.cc:1403] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.068s user 0.017s sys 0.008s
I20260317 21:18:55.226925 7477 ts_tablet_manager.cc:1916] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.228328 7462 raft_consensus.cc:359] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.229122 7462 raft_consensus.cc:385] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.229377 7462 raft_consensus.cc:740] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.229965 7462 consensus_queue.cc:260] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.231336 7459 tablet_bootstrap.cc:492] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:55.234290 7462 ts_tablet_manager.cc:1434] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.009s user 0.002s sys 0.003s
I20260317 21:18:55.235127 7462 tablet_bootstrap.cc:492] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.248775 7462 tablet_bootstrap.cc:654] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.253724 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 99b954ac41134300823458a7a25ef738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.253979 7304 tablet_service.cc:1511] Processing CreateTablet for tablet e4522181cc434203ac14c9b07587fec8 (DEFAULT_TABLE table=default.Table_34196bd319d74982a3e5f88cb0957c14 [id=577b3aea931d4e518d588b81917b49c8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.255285 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e4522181cc434203ac14c9b07587fec8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.258360 7467 ts_tablet_manager.cc:1434] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.250s user 0.003s sys 0.006s
I20260317 21:18:55.259370 7462 tablet_bootstrap.cc:492] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.260336 7154 tablet_service.cc:1511] Processing CreateTablet for tablet dc929bae186841d2aba674659e457301 (DEFAULT_TABLE table=default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.261345 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc929bae186841d2aba674659e457301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.270401 7528 raft_consensus.cc:515] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.273691 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "33f4e1fbb0d24813b0758cae9477f006" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:55.273980 7528 leader_election.cc:290] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:55.274291 7183 raft_consensus.cc:3060] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:55.275177 7549 raft_consensus.cc:493] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.275648 7549 raft_consensus.cc:515] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:55.277086 6973 leader_election.cc:343] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:55.277889 7549 leader_election.cc:290] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:55.279696 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99b954ac41134300823458a7a25ef738" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:55.281025 7580 raft_consensus.cc:493] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.281821 7462 ts_tablet_manager.cc:1403] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.047s user 0.007s sys 0.004s
I20260317 21:18:55.281502 7580 raft_consensus.cc:515] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.283519 7580 leader_election.cc:290] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:55.285450 7261 leader_election.cc:343] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.286880 7183 raft_consensus.cc:2468] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:18:55.277945 7264 consensus_peers.cc:597] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:55.288324 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:55.289041 7328 raft_consensus.cc:2468] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:55.285450 7462 raft_consensus.cc:359] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.294868 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_5f4dc71734d540319e6f0d19d4037ba8" } modify_external_catalogs: true
I20260317 21:18:55.295202 7462 raft_consensus.cc:385] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.295423 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_5f4dc71734d540319e6f0d19d4037ba8" } modify_external_catalogs: true
I20260317 21:18:55.295460 7462 raft_consensus.cc:740] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.296613 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_8e6a77924a3748cbbb655ae7c443cfb4" } new_table_name: "default.Table_713bafc801d244d1a80d9157aebd581b" modify_external_catalogs: true
I20260317 21:18:55.298070 7462 consensus_queue.cc:260] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.291338 7119 leader_election.cc:304] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:55.300832 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1885223af9524a43b0d34db30e68e6f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:55.302784 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1885223af9524a43b0d34db30e68e6f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:55.303040 7580 raft_consensus.cc:2804] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:55.303686 7580 raft_consensus.cc:493] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:55.303756 7462 ts_tablet_manager.cc:1434] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.022s user 0.004s sys 0.004s
I20260317 21:18:55.304019 7580 raft_consensus.cc:3060] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:55.304720 7462 tablet_bootstrap.cc:492] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.309440 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99b954ac41134300823458a7a25ef738" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:55.310807 7462 tablet_bootstrap.cc:654] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.310885 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:55.315008 6971 leader_election.cc:304] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
W20260317 21:18:55.315598 7120 leader_election.cc:343] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.316244 7528 raft_consensus.cc:2804] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
W20260317 21:18:55.318126 7264 leader_election.cc:343] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.318606 7264 leader_election.cc:304] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.319759 7549 raft_consensus.cc:2749] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:55.325721 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_45189d08af7842588601921563fa3b41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:55.312186 7580 raft_consensus.cc:515] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:55.328812 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45189d08af7842588601921563fa3b41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:55.332758 7528 raft_consensus.cc:697] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:55.333071 7462 tablet_bootstrap.cc:492] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.333379 7459 tablet_bootstrap.cc:654] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.333473 7462 ts_tablet_manager.cc:1403] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.000s
I20260317 21:18:55.334760 7467 tablet_bootstrap.cc:492] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:55.335902 7462 raft_consensus.cc:359] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.336535 7462 raft_consensus.cc:385] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.336844 7462 raft_consensus.cc:740] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.338044 7580 leader_election.cc:290] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:55.342375 7462 consensus_queue.cc:260] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.345832 7462 ts_tablet_manager.cc:1434] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260317 21:18:55.346637 7462 tablet_bootstrap.cc:492] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.356181 7462 tablet_bootstrap.cc:654] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.265864 7478 ts_tablet_manager.cc:1929] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20260317 21:18:55.361968 7478 log.cc:1199] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/f41f18434a744b03b6ff53f3d7760e55
I20260317 21:18:55.362914 7478 ts_tablet_manager.cc:1950] T f41f18434a744b03b6ff53f3d7760e55 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:55.365052 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 66) took 1464 ms. Trace:
I20260317 21:18:55.369465 7462 tablet_bootstrap.cc:492] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.333910 7528 consensus_queue.cc:237] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.371512 7462 ts_tablet_manager.cc:1403] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.000s
I20260317 21:18:55.371891 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:55.374034 7462 raft_consensus.cc:359] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:55.375280 6662 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_34196bd319d74982a3e5f88cb0957c14 [id=577b3aea931d4e518d588b81917b49c8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e4522181cc434203ac14c9b07587fec8 consensus state has no leader
I20260317 21:18:55.378628 7462 raft_consensus.cc:385] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.379436 7462 raft_consensus.cc:740] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.380558 7462 consensus_queue.cc:260] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.385380 7462 ts_tablet_manager.cc:1434] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260317 21:18:55.388229 7462 tablet_bootstrap.cc:492] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.398416 7462 tablet_bootstrap.cc:654] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.400672 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:18:55.402451 6666 catalog_manager.cc:6741] ReplaceTablet: tablet b6fb7cb082cf47ce8cf2a98146633af7 deleted and replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7
W20260317 21:18:55.405017 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4522181cc434203ac14c9b07587fec8 consensus state has no leader
I20260317 21:18:55.408797 6673 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 27977b00ae1a4e7ca2eddaa693d11a65
I20260317 21:18:55.411289 6673 catalog_manager.cc:6741] ReplaceTablet: tablet 27977b00ae1a4e7ca2eddaa693d11a65 deleted and replaced by tablet 039d8bed256a43a6b98075b8430cbf3b
W20260317 21:18:55.374217 7120 leader_election.cc:343] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.415536 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 27977b00ae1a4e7ca2eddaa693d11a65 with delete_type TABLET_DATA_DELETED (replaced by tablet 039d8bed256a43a6b98075b8430cbf3b) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:55.416009 7012 tablet_service.cc:1558] Processing DeleteTablet for tablet f0d55550bf914019b83e83fa4ffcb64b with delete_type TABLET_DATA_DELETED (replaced by tablet e4522181cc434203ac14c9b07587fec8) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:55.416496 7462 tablet_bootstrap.cc:492] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.416795 7478 rpcz_store.cc:276] 0317 21:18:53.900737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:53.908996 (+ 8259us) service_pool.cc:225] Handling call
0317 21:18:54.956169 (+1047173us) ts_tablet_manager.cc:1151] Deleting tablet f41f18434a744b03b6ff53f3d7760e55
0317 21:18:54.956210 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.088751 (+132541us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.265640 (+176889us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.364985 (+ 99345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1046497}
I20260317 21:18:55.418259 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet f41f18434a744b03b6ff53f3d7760e55 (table default.Table_8e6a77924a3748cbbb655ae7c443cfb4 [id=6de336aaef3744f58b4fdfab525c8f2f]) successfully deleted
I20260317 21:18:55.419263 7478 tablet_replica.cc:333] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:55.419740 7012 tablet_service.cc:1558] Processing DeleteTablet for tablet 27977b00ae1a4e7ca2eddaa693d11a65 with delete_type TABLET_DATA_DELETED (replaced by tablet 039d8bed256a43a6b98075b8430cbf3b) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:55.420028 7478 raft_consensus.cc:2243] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:55.419111 7462 ts_tablet_manager.cc:1403] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.002s
I20260317 21:18:55.423266 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:55.423662 7462 raft_consensus.cc:359] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.423960 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:55.424305 7462 raft_consensus.cc:385] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.428071 7467 tablet_bootstrap.cc:654] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.424625 7462 raft_consensus.cc:740] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.420457 7478 raft_consensus.cc:2272] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.428982 7477 ts_tablet_manager.cc:1929] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.429533 7462 consensus_queue.cc:260] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.431007 7478 ts_tablet_manager.cc:1916] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.436908 7462 ts_tablet_manager.cc:1434] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260317 21:18:55.437747 7462 tablet_bootstrap.cc:492] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.440987 7459 tablet_bootstrap.cc:492] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:55.441478 7459 ts_tablet_manager.cc:1403] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.210s user 0.011s sys 0.000s
I20260317 21:18:55.444021 7462 tablet_bootstrap.cc:654] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.429351 7477 log.cc:1199] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/3360925bb5e2415db39a551800ca28d0
I20260317 21:18:55.444775 7477 ts_tablet_manager.cc:1950] T 3360925bb5e2415db39a551800ca28d0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:55.448786 7459 raft_consensus.cc:359] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:55.449702 7459 raft_consensus.cc:385] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.451387 7477 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 63) took 1389 ms. Trace:
I20260317 21:18:55.451763 7477 rpcz_store.cc:276] 0317 21:18:54.062206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.062366 (+ 160us) service_pool.cc:225] Handling call
0317 21:18:55.173822 (+1111456us) ts_tablet_manager.cc:1151] Deleting tablet 3360925bb5e2415db39a551800ca28d0
0317 21:18:55.201472 (+ 27650us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.372748 (+171276us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.428718 (+ 55970us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.451316 (+ 22598us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4647,"tablet-delete.queue_time_us":1110829}
I20260317 21:18:55.453079 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 3360925bb5e2415db39a551800ca28d0 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]) successfully deleted
I20260317 21:18:55.453570 6669 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_cae455ad5e214610bcfbeff2d7054eff" } modify_external_catalogs: true
I20260317 21:18:55.454183 6669 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_cae455ad5e214610bcfbeff2d7054eff" } modify_external_catalogs: true
I20260317 21:18:55.457173 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bfc5d1e1d8fa42cf920a94eab51746b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:55.460162 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bfc5d1e1d8fa42cf920a94eab51746b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:55.462109 7462 tablet_bootstrap.cc:492] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.462512 7462 ts_tablet_manager.cc:1403] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.006s
I20260317 21:18:55.465327 7462 raft_consensus.cc:359] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.465823 7477 tablet_replica.cc:333] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:55.465965 7462 raft_consensus.cc:385] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.466204 7462 raft_consensus.cc:740] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.466506 7477 raft_consensus.cc:2243] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:55.466943 7477 raft_consensus.cc:2272] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.466758 7462 consensus_queue.cc:260] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.475191 7462 ts_tablet_manager.cc:1434] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.012s user 0.006s sys 0.001s
I20260317 21:18:55.476115 7462 tablet_bootstrap.cc:492] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.468578 7477 ts_tablet_manager.cc:1916] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.450415 7459 raft_consensus.cc:740] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.480787 7459 consensus_queue.cc:260] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:55.493975 7467 tablet_bootstrap.cc:492] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:55.494493 7467 ts_tablet_manager.cc:1403] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.160s user 0.011s sys 0.000s
I20260317 21:18:55.502763 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet dc929bae186841d2aba674659e457301
I20260317 21:18:55.505491 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet dc929bae186841d2aba674659e457301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:55.519038 7459 ts_tablet_manager.cc:1434] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.077s user 0.003s sys 0.003s
W20260317 21:18:55.522279 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd7cf19121b5496a849118de4a29db5b has no consensus state
I20260317 21:18:55.526419 7467 raft_consensus.cc:359] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.527108 7467 raft_consensus.cc:385] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.526990 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet dc929bae186841d2aba674659e457301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:55.527343 7467 raft_consensus.cc:740] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.549940 7477 ts_tablet_manager.cc:1929] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.550760 7477 log.cc:1199] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/3ef4a015f5c14ee6afd045a87eaa6a89
I20260317 21:18:55.552024 7477 ts_tablet_manager.cc:1950] T 3ef4a015f5c14ee6afd045a87eaa6a89 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:55.565328 7477 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 64) took 1361 ms. Trace:
I20260317 21:18:55.572814 7477 rpcz_store.cc:276] 0317 21:18:54.203931 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.206066 (+ 2135us) service_pool.cc:225] Handling call
0317 21:18:55.465671 (+1259605us) ts_tablet_manager.cc:1151] Deleting tablet 3ef4a015f5c14ee6afd045a87eaa6a89
0317 21:18:55.465713 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.501045 (+ 35332us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.549521 (+ 48476us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.565255 (+ 15734us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1258896}
I20260317 21:18:55.574380 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 3ef4a015f5c14ee6afd045a87eaa6a89 (table default.Table_f3adbc8bf1994597bf7be6a6792ebf5a [id=16fb03a1fcfb4eff833f8ff6292c0b21]) successfully deleted
I20260317 21:18:55.579027 7477 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 66) took 1254 ms. Trace:
I20260317 21:18:55.580750 7477 rpcz_store.cc:276] 0317 21:18:54.325003 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.352769 (+ 27766us) service_pool.cc:225] Handling call
0317 21:18:55.574443 (+1221674us) ts_tablet_manager.cc:1151] Deleting tablet 291e2bee709243f09b338d99edbe0cb1
0317 21:18:55.574488 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.578958 (+ 4470us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1216085}
W20260317 21:18:55.586086 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 291e2bee709243f09b338d99edbe0cb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 291e2bee709243f09b338d99edbe0cb1 already in progress: creating tablet
I20260317 21:18:55.569936 7459 tablet_bootstrap.cc:492] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:55.587380 7477 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 70) took 1105 ms. Trace:
I20260317 21:18:55.588047 7477 rpcz_store.cc:276] 0317 21:18:54.481760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.484874 (+ 3114us) service_pool.cc:225] Handling call
0317 21:18:55.587054 (+1102180us) ts_tablet_manager.cc:1151] Deleting tablet b196f1e1572f4a62b45df7ffc581cd7a
0317 21:18:55.587092 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.587326 (+ 234us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1101223}
I20260317 21:18:55.527926 7467 consensus_queue.cc:260] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:55.589526 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet b196f1e1572f4a62b45df7ffc581cd7a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b196f1e1572f4a62b45df7ffc581cd7a already in progress: creating tablet
W20260317 21:18:55.591426 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet c6d5385171364a078d968bb0a489383a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d5385171364a078d968bb0a489383a already in progress: creating tablet
W20260317 21:18:55.593189 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 163cc8c04ca94a269c7ae2efaa162525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163cc8c04ca94a269c7ae2efaa162525 already in progress: creating tablet
I20260317 21:18:55.544520 7462 tablet_bootstrap.cc:654] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.595633 7477 tablet_replica.cc:333] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:55.606137 7477 raft_consensus.cc:2243] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:55.607402 7477 pending_rounds.cc:70] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:18:55.607707 7477 pending_rounds.cc:77] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412445689335808 op_type: NO_OP noop_request { }
I20260317 21:18:55.608158 7477 raft_consensus.cc:2889] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:55.608583 7477 raft_consensus.cc:2272] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.611162 7477 ts_tablet_manager.cc:1916] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.615821 7304 tablet_service.cc:1511] Processing CreateTablet for tablet dc929bae186841d2aba674659e457301 (DEFAULT_TABLE table=default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:55.456131 7483 ts_tablet_manager.cc:1929] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.617322 7483 log.cc:1199] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/b196f1e1572f4a62b45df7ffc581cd7a
I20260317 21:18:55.617575 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc929bae186841d2aba674659e457301. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:55.617560 7467 ts_tablet_manager.cc:1434] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.123s user 0.006s sys 0.000s
I20260317 21:18:55.617695 7656 raft_consensus.cc:493] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.618425 7483 ts_tablet_manager.cc:1950] T b196f1e1572f4a62b45df7ffc581cd7a P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:55.618511 7656 raft_consensus.cc:515] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.620214 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.621076 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.622016 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 27977b00ae1a4e7ca2eddaa693d11a65 with delete_type TABLET_DATA_DELETED (replaced by tablet 039d8bed256a43a6b98075b8430cbf3b) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.543161 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:55.622691 7328 raft_consensus.cc:3060] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:55.622859 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet dc929bae186841d2aba674659e457301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:55.623970 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d8df008b6b284340ab0c3a41102c096f" } new_table_name: "default.Table_b451641b2795486b8d68f72ab543b5b2" modify_external_catalogs: true
I20260317 21:18:55.628927 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_aef4da11f840465eb7fc80e6423d7454"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:55.630671 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aef4da11f840465eb7fc80e6423d7454 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:55.631765 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 73) took 1136 ms. Trace:
I20260317 21:18:55.632223 7483 rpcz_store.cc:276] 0317 21:18:54.495513 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.508799 (+ 13286us) service_pool.cc:225] Handling call
0317 21:18:55.043091 (+534292us) ts_tablet_manager.cc:1151] Deleting tablet b196f1e1572f4a62b45df7ffc581cd7a
0317 21:18:55.043133 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.068377 (+ 25244us) spinlock_profiling.cc:243] Waited 20.2 ms on lock 0x7b5400021a54. stack: 00007f8cd9137423 00007f8cd9137320 00007f8cd8cac574 0000000000509544 00000000005094d9 00007f8ce01927a1 00007f8ce0231292 00007f8ce065357c 00007f8ce12aa8aa 00007f8ce12db510 00007f8ce12c1add 00007f8ce12c1a89 00007f8ce12c1a19 00007f8ce12c19e1 00007f8ce12c0c0d 00000000005ab084
0317 21:18:55.287801 (+219424us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.455710 (+167909us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.631576 (+175866us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":46552448,"tablet-delete.queue_time_us":533356}
I20260317 21:18:55.634600 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 75) took 1088 ms. Trace:
I20260317 21:18:55.634956 7483 rpcz_store.cc:276] 0317 21:18:54.546278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.546437 (+ 159us) service_pool.cc:225] Handling call
0317 21:18:55.634099 (+1087662us) ts_tablet_manager.cc:1151] Deleting tablet c6d5385171364a078d968bb0a489383a
0317 21:18:55.634188 (+ 89us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.634537 (+ 349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1086891}
I20260317 21:18:55.636018 7483 tablet_replica.cc:333] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:55.639303 7657 raft_consensus.cc:493] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.641830 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet b196f1e1572f4a62b45df7ffc581cd7a (table default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3]) successfully deleted
I20260317 21:18:55.642269 7656 leader_election.cc:290] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:18:55.642527 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet c6d5385171364a078d968bb0a489383a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d5385171364a078d968bb0a489383a already in progress: creating tablet
I20260317 21:18:55.642969 7657 raft_consensus.cc:515] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.650719 7657 leader_election.cc:290] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:55.650995 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:55.654557 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 564a02888d854ea58d7a590cba816b86 has no consensus state
I20260317 21:18:55.658833 7656 raft_consensus.cc:493] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.659305 7656 raft_consensus.cc:515] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.661322 7656 leader_election.cc:290] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:55.628120 7528 raft_consensus.cc:493] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.644125 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:55.657133 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1094068903840a8ab5d07c7fb0635bf" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:55.663127 7528 raft_consensus.cc:515] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } }
I20260317 21:18:55.665118 7528 leader_election.cc:290] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:18:55.667088 6971 leader_election.cc:343] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: f0d55550bf914019b83e83fa4ffcb64b
I20260317 21:18:55.671566 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:55.679090 7467 tablet_bootstrap.cc:492] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:55.663817 7657 raft_consensus.cc:493] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.683290 7657 raft_consensus.cc:515] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.686571 6675 master_service.cc:954] ReplaceTablet: received request to replace tablet 27977b00ae1a4e7ca2eddaa693d11a65 from {username='slave'} at 127.0.0.1:39038
W20260317 21:18:55.670188 7264 leader_election.cc:343] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.670176 7549 consensus_queue.cc:1048] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:18:55.689790 7264 leader_election.cc:343] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.689994 7657 leader_election.cc:290] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:55.691020 7459 tablet_bootstrap.cc:654] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.648159 7328 raft_consensus.cc:2468] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:55.679245 7483 raft_consensus.cc:2243] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260317 21:18:55.690420 7264 leader_election.cc:343] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.694006 7467 tablet_bootstrap.cc:654] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.694494 7483 raft_consensus.cc:2272] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.695399 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53dd9f8a1ccc47f884bfba99781aed16" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:55.690897 7261 leader_election.cc:343] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.697167 7483 ts_tablet_manager.cc:1916] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.699918 7261 leader_election.cc:304] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.700773 7657 raft_consensus.cc:2749] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:55.701152 7119 leader_election.cc:304] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.702550 7580 raft_consensus.cc:2804] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:55.704458 7467 tablet_bootstrap.cc:492] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:55.704941 7467 ts_tablet_manager.cc:1403] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.026s user 0.009s sys 0.003s
W20260317 21:18:55.705660 7264 leader_election.cc:343] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.707103 7467 raft_consensus.cc:359] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.707696 7467 raft_consensus.cc:385] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.707930 7467 raft_consensus.cc:740] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.710430 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1094068903840a8ab5d07c7fb0635bf" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:55.710377 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_05b675a3f5f649f0b419a59478fd907a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:55.710944 7459 tablet_bootstrap.cc:492] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:55.711349 7459 ts_tablet_manager.cc:1403] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.142s user 0.007s sys 0.006s
I20260317 21:18:55.713459 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:55.713810 7459 raft_consensus.cc:359] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.714454 7459 raft_consensus.cc:385] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.714829 7459 raft_consensus.cc:740] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.715765 7459 consensus_queue.cc:260] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.721714 7478 ts_tablet_manager.cc:1929] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.724871 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet 8ba0874a8cd746e9a33596ea252b3b14 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:55.725457 7459 ts_tablet_manager.cc:1434] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.014s user 0.006s sys 0.001s
I20260317 21:18:55.725875 6673 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e4715bec3faa471cbdcb63c57766180a" } modify_external_catalogs: true
I20260317 21:18:55.726444 6673 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e4715bec3faa471cbdcb63c57766180a" } modify_external_catalogs: true
I20260317 21:18:55.727679 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_49f59ddc9db844c5bcfb828a4735238b" } new_table_name: "default.Table_411f7f4c71494d26b681f5e76fa36214" modify_external_catalogs: true
I20260317 21:18:55.730597 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 291e2bee709243f09b338d99edbe0cb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:53 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:55.738396 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_cf3bed4cf36c4e05a6ca6ab34245a067"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:55.740453 7462 tablet_bootstrap.cc:492] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
W20260317 21:18:55.743572 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_05b675a3f5f649f0b419a59478fd907a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:55.743631 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf3bed4cf36c4e05a6ca6ab34245a067 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:55.748479 7478 log.cc:1199] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:55.749327 7478 ts_tablet_manager.cc:1950] T 33f4e1fbb0d24813b0758cae9477f006 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:55.749897 7462 ts_tablet_manager.cc:1403] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.267s user 0.014s sys 0.020s
I20260317 21:18:55.726125 7467 consensus_queue.cc:260] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.748096 7459 tablet_bootstrap.cc:492] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:55.762180 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 67) took 1729 ms. Trace:
I20260317 21:18:55.762938 7462 raft_consensus.cc:359] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.763649 7462 raft_consensus.cc:385] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.763942 7462 raft_consensus.cc:740] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.768857 7478 rpcz_store.cc:276] 0317 21:18:54.032826 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.032987 (+ 161us) service_pool.cc:225] Handling call
0317 21:18:55.419120 (+1386133us) ts_tablet_manager.cc:1151] Deleting tablet 33f4e1fbb0d24813b0758cae9477f006
0317 21:18:55.419163 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.564100 (+144937us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.721494 (+157394us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.762110 (+ 40616us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1384708}
I20260317 21:18:55.782042 7478 tablet_replica.cc:333] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:55.794036 7467 ts_tablet_manager.cc:1434] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.089s user 0.008s sys 0.000s
I20260317 21:18:55.793430 7462 consensus_queue.cc:260] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.795192 7478 raft_consensus.cc:2243] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:55.795468 7483 ts_tablet_manager.cc:1929] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.795768 7478 raft_consensus.cc:2272] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.798174 7467 tablet_bootstrap.cc:492] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:55.800830 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "53dd9f8a1ccc47f884bfba99781aed16" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:55.803015 7478 ts_tablet_manager.cc:1916] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:18:55.803886 6973 leader_election.cc:343] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: f0d55550bf914019b83e83fa4ffcb64b
I20260317 21:18:55.804317 6973 leader_election.cc:304] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:55.805330 7528 raft_consensus.cc:2749] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:55.809046 7483 log.cc:1199] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/163cc8c04ca94a269c7ae2efaa162525
I20260317 21:18:55.810976 7483 ts_tablet_manager.cc:1950] T 163cc8c04ca94a269c7ae2efaa162525 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:55.813897 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 77) took 1215 ms. Trace:
I20260317 21:18:55.817096 7483 rpcz_store.cc:276] 0317 21:18:54.598317 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.599111 (+ 794us) service_pool.cc:225] Handling call
0317 21:18:55.635837 (+1036726us) ts_tablet_manager.cc:1151] Deleting tablet 163cc8c04ca94a269c7ae2efaa162525
0317 21:18:55.635905 (+ 68us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.759700 (+123795us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.793099 (+ 33399us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.813800 (+ 20701us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":38986,"tablet-delete.queue_time_us":1035541}
I20260317 21:18:55.819150 7459 tablet_bootstrap.cc:654] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.821940 7580 raft_consensus.cc:697] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:55.822942 7580 consensus_queue.cc:237] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.804086 7462 ts_tablet_manager.cc:1434] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.054s user 0.010s sys 0.002s
W20260317 21:18:55.806321 7261 leader_election.cc:343] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.826264 7261 leader_election.cc:304] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.826673 7462 tablet_bootstrap.cc:492] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.827198 7657 raft_consensus.cc:2749] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:55.828683 7261 leader_election.cc:343] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.829193 7261 leader_election.cc:304] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.830480 7657 raft_consensus.cc:2749] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:55.831642 7261 leader_election.cc:343] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.829438 7477 ts_tablet_manager.cc:1929] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:55.831637 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 163cc8c04ca94a269c7ae2efaa162525 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]) successfully deleted
I20260317 21:18:55.832177 7261 leader_election.cc:304] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.833163 7657 raft_consensus.cc:2749] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:55.833563 7483 tablet_replica.cc:333] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:55.834318 7483 raft_consensus.cc:2243] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:55.833127 7477 log.cc:1199] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/33f4e1fbb0d24813b0758cae9477f006
I20260317 21:18:55.836247 7483 raft_consensus.cc:2272] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.778904 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 33f4e1fbb0d24813b0758cae9477f006 (table default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]) successfully deleted
W20260317 21:18:55.839217 6659 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 60355.1us
I20260317 21:18:55.840517 7483 ts_tablet_manager.cc:1916] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.844225 7528 raft_consensus.cc:493] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.838706 7462 tablet_bootstrap.cc:654] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:55.844975 7528 raft_consensus.cc:515] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.846376 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_351b690e266140a3a403e918f71f9d69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:55.846989 7528 leader_election.cc:290] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:55.847980 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_351b690e266140a3a403e918f71f9d69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:55.849205 7477 ts_tablet_manager.cc:1950] T 33f4e1fbb0d24813b0758cae9477f006 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:55.835635 7467 tablet_bootstrap.cc:654] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:55.837908 7120 consensus_peers.cc:597] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:55.848551 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "291e2bee709243f09b338d99edbe0cb1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:55.851128 6971 leader_election.cc:343] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:55.852429 7657 raft_consensus.cc:493] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.852909 7657 raft_consensus.cc:515] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.854674 7657 leader_election.cc:290] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:55.856048 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ba0874a8cd746e9a33596ea252b3b14" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:55.857687 7264 leader_election.cc:343] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.861460 7477 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 75) took 1152 ms. Trace:
I20260317 21:18:55.861900 7477 rpcz_store.cc:276] 0317 21:18:54.703942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.718614 (+ 14672us) service_pool.cc:225] Handling call
0317 21:18:55.595477 (+876863us) ts_tablet_manager.cc:1151] Deleting tablet 33f4e1fbb0d24813b0758cae9477f006
0317 21:18:55.595517 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.730122 (+134605us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.824636 (+ 94514us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.856606 (+ 31970us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":763355}
I20260317 21:18:55.863504 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 33f4e1fbb0d24813b0758cae9477f006 (table default.Table_844e7186ad624a5da968ffef9aac1610 [id=bfefa5ba10fa49409881d9b0115f6bdc]) successfully deleted
W20260317 21:18:55.865932 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 53dd9f8a1ccc47f884bfba99781aed16 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53dd9f8a1ccc47f884bfba99781aed16 already in progress: creating tablet
I20260317 21:18:55.866142 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ba0874a8cd746e9a33596ea252b3b14" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:55.867962 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_16cf89af28ec48e59cac714020a22b8b" } new_table_name: "default.Table_222217e8357d431baa8615150079a999" modify_external_catalogs: true
I20260317 21:18:55.868446 7477 tablet_replica.cc:333] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:55.869207 7477 raft_consensus.cc:2243] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:55.871605 7459 tablet_bootstrap.cc:492] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:55.871886 7477 raft_consensus.cc:2272] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:55.872022 7459 ts_tablet_manager.cc:1403] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.124s user 0.011s sys 0.006s
I20260317 21:18:55.874447 7459 raft_consensus.cc:359] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.875146 7459 raft_consensus.cc:385] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.876740 7477 ts_tablet_manager.cc:1916] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:55.879097 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_fcec6e585d6245ff828c8c0c98450d63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:55.880903 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fcec6e585d6245ff828c8c0c98450d63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:55.887718 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 935faf7f9129440a91635bfa96191cd6 consensus state has no leader
I20260317 21:18:55.893134 7478 ts_tablet_manager.cc:1929] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.875396 7459 raft_consensus.cc:740] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.895931 7462 tablet_bootstrap.cc:492] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:55.896476 7462 ts_tablet_manager.cc:1403] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.070s user 0.009s sys 0.001s
I20260317 21:18:55.899035 7462 raft_consensus.cc:359] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:55.899333 7261 leader_election.cc:343] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.899828 7261 leader_election.cc:304] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.900718 7656 raft_consensus.cc:2749] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:55.902079 6973 leader_election.cc:343] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 291e2bee709243f09b338d99edbe0cb1
I20260317 21:18:55.902498 6973 leader_election.cc:304] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:55.903445 7528 raft_consensus.cc:2749] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:55.894171 7459 consensus_queue.cc:260] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.917127 6674 master_service.cc:954] ReplaceTablet: received request to replace tablet 82f8dd260a134810a50465e4a77848ca from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:55.919175 7459 ts_tablet_manager.cc:1434] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.047s user 0.002s sys 0.005s
I20260317 21:18:55.903373 7467 tablet_bootstrap.cc:492] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:55.919828 7467 ts_tablet_manager.cc:1403] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.122s user 0.011s sys 0.002s
I20260317 21:18:55.922451 7467 raft_consensus.cc:359] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.923951 7467 raft_consensus.cc:385] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.924037 7462 raft_consensus.cc:385] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:55.924269 7467 raft_consensus.cc:740] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.924398 7462 raft_consensus.cc:740] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:55.925123 7467 consensus_queue.cc:260] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.927155 7467 ts_tablet_manager.cc:1434] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.007s user 0.002s sys 0.003s
I20260317 21:18:55.931553 7462 consensus_queue.cc:260] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.932195 7478 log.cc:1199] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/291e2bee709243f09b338d99edbe0cb1
I20260317 21:18:55.934449 7478 ts_tablet_manager.cc:1950] T 291e2bee709243f09b338d99edbe0cb1 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:55.929440 7036 raft_consensus.cc:3060] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:55.942588 7459 tablet_bootstrap.cc:492] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:55.934826 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:55.948158 7580 consensus_queue.cc:1048] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:55.938257 7467 tablet_bootstrap.cc:492] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:55.953004 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 69) took 1669 ms. Trace:
I20260317 21:18:55.954670 7478 rpcz_store.cc:276] 0317 21:18:54.284013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.284238 (+ 225us) service_pool.cc:225] Handling call
0317 21:18:55.780215 (+1495977us) ts_tablet_manager.cc:1151] Deleting tablet 291e2bee709243f09b338d99edbe0cb1
0317 21:18:55.781187 (+ 972us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.838941 (+ 57754us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.892907 (+ 53966us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:55.952935 (+ 60028us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":8773,"tablet-delete.queue_time_us":1494949}
I20260317 21:18:55.953234 7580 raft_consensus.cc:493] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:55.959220 7580 raft_consensus.cc:515] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:55.961853 7462 ts_tablet_manager.cc:1434] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.065s user 0.006s sys 0.000s
I20260317 21:18:55.963267 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:55.965744 7120 leader_election.cc:343] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:55.967806 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:55.969172 7328 raft_consensus.cc:2468] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:55.970510 7119 leader_election.cc:304] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:55.972061 7662 raft_consensus.cc:2804] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:55.972419 7662 raft_consensus.cc:493] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:55.972913 7662 raft_consensus.cc:3060] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:55.973881 7462 tablet_bootstrap.cc:492] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:55.977166 7477 ts_tablet_manager.cc:1929] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:55.987429 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1d14e25fb3ec49bfb39f46819cdc1d5b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:55.990283 7477 log.cc:1199] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/f0d55550bf914019b83e83fa4ffcb64b
I20260317 21:18:55.993841 7477 ts_tablet_manager.cc:1950] T f0d55550bf914019b83e83fa4ffcb64b P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:55.994958 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b1de6d27253e4eb8b308dc7f2fc0b643"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:55.998901 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1d14e25fb3ec49bfb39f46819cdc1d5b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.000051 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 291e2bee709243f09b338d99edbe0cb1 (table default.Table_2fab17210bfa496ba34bb9580bee9d79 [id=040d2e5e61ef41e4ac8953a3d63a226c]) successfully deleted
W20260317 21:18:56.000047 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.004020 7483 ts_tablet_manager.cc:1929] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:56.005918 7478 tablet_replica.cc:333] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:56.006546 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet f0d55550bf914019b83e83fa4ffcb64b (table default.Table_24103dddfd444503aa58762e5d598637 [id=577b3aea931d4e518d588b81917b49c8]) successfully deleted
I20260317 21:18:56.008834 7483 log.cc:1199] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:18:56.009825 7483 ts_tablet_manager.cc:1950] T 53dd9f8a1ccc47f884bfba99781aed16 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:56.017062 7477 tablet_replica.cc:333] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:56.008263 7462 tablet_bootstrap.cc:654] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.017908 7477 raft_consensus.cc:2243] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.018486 7477 raft_consensus.cc:2272] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.020515 7478 raft_consensus.cc:2243] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:56.020630 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 81) took 1083 ms. Trace:
I20260317 21:18:56.022836 7478 pending_rounds.cc:70] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:18:56.020511 7477 ts_tablet_manager.cc:1916] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.022336 7483 rpcz_store.cc:276] 0317 21:18:54.936999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.940804 (+ 3805us) service_pool.cc:225] Handling call
0317 21:18:55.833392 (+892588us) ts_tablet_manager.cc:1151] Deleting tablet 53dd9f8a1ccc47f884bfba99781aed16
0317 21:18:55.833455 (+ 63us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:55.922340 (+ 88885us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.003794 (+ 81454us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.020564 (+ 16770us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":891787}
I20260317 21:18:56.025439 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 53dd9f8a1ccc47f884bfba99781aed16 (table default.Table_afb9c6074ee54f86a4d029cf8cdbb825 [id=dd080d09b8324bcabbe022d42352fadb]) successfully deleted
I20260317 21:18:56.025548 7549 raft_consensus.cc:493] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.025959 7549 raft_consensus.cc:515] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.026947 7478 pending_rounds.cc:77] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412444452093952 op_type: NO_OP noop_request { }
I20260317 21:18:56.027333 7478 raft_consensus.cc:2889] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:56.027644 7478 raft_consensus.cc:2272] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.009326 7662 raft_consensus.cc:515] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.031651 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:55.994201 7580 leader_election.cc:290] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.033233 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f980bd133da4b04b101e8df99ccca6f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:56.034448 7120 leader_election.cc:343] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:56.034873 7264 leader_election.cc:343] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.035543 7549 leader_election.cc:290] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:56.037333 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f980bd133da4b04b101e8df99ccca6f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:56.038647 7261 leader_election.cc:343] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.039077 7261 leader_election.cc:304] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.040117 7549 raft_consensus.cc:2749] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.023909 7467 tablet_bootstrap.cc:654] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.053563 7459 tablet_bootstrap.cc:654] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.054742 7483 tablet_replica.cc:333] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:56.055533 7483 raft_consensus.cc:2243] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.056051 7483 raft_consensus.cc:2272] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.056555 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:56.057112 7328 raft_consensus.cc:3060] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:56.058390 7478 ts_tablet_manager.cc:1916] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.065481 7328 raft_consensus.cc:2468] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:56.067056 7119 leader_election.cc:304] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.068348 7580 raft_consensus.cc:2804] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:56.068944 7580 raft_consensus.cc:697] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:56.073887 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0a38822238d94d4eaadded2ca39840f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:56.075795 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a38822238d94d4eaadded2ca39840f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.070029 7580 consensus_queue.cc:237] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.075857 7662 leader_election.cc:290] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.085639 7661 consensus_queue.cc:1048] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:56.077862 7477 ts_tablet_manager.cc:1929] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:56.065663 7483 ts_tablet_manager.cc:1916] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.092911 7477 log.cc:1199] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/27977b00ae1a4e7ca2eddaa693d11a65
I20260317 21:18:56.093711 7477 ts_tablet_manager.cc:1950] T 27977b00ae1a4e7ca2eddaa693d11a65 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:56.096596 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_8cba5340b2044350864c86b46e1a9488" } new_table_name: "default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f" modify_external_catalogs: true
I20260317 21:18:56.099952 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 27977b00ae1a4e7ca2eddaa693d11a65 (table default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]) successfully deleted
I20260317 21:18:56.110251 7462 tablet_bootstrap.cc:492] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:56.110669 7462 ts_tablet_manager.cc:1403] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.137s user 0.007s sys 0.004s
I20260317 21:18:56.113628 7462 raft_consensus.cc:359] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.114295 7462 raft_consensus.cc:385] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.114550 7462 raft_consensus.cc:740] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:18:56.115180 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet b6fb7cb082cf47ce8cf2a98146633af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fb7cb082cf47ce8cf2a98146633af7 already in progress: creating tablet
I20260317 21:18:56.115114 7462 consensus_queue.cc:260] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.118546 7462 ts_tablet_manager.cc:1434] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260317 21:18:56.120270 7462 tablet_bootstrap.cc:492] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:18:56.120260 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet dc929bae186841d2aba674659e457301 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc929bae186841d2aba674659e457301 already in progress: creating tablet
I20260317 21:18:56.123344 7467 tablet_bootstrap.cc:492] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:56.123786 7467 ts_tablet_manager.cc:1403] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.186s user 0.009s sys 0.001s
I20260317 21:18:56.124311 7477 tablet_replica.cc:333] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:56.125415 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_aa8ccdaa054d482c934eb9db7cfaf5b0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:56.127493 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa8ccdaa054d482c934eb9db7cfaf5b0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.137137 7477 raft_consensus.cc:2243] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.140923 7477 raft_consensus.cc:2272] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.147064 7459 tablet_bootstrap.cc:492] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:56.147581 7459 ts_tablet_manager.cc:1403] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.205s user 0.012s sys 0.000s
I20260317 21:18:56.150086 7459 raft_consensus.cc:359] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.156566 7467 raft_consensus.cc:359] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.157478 7467 raft_consensus.cc:385] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.164817 7459 raft_consensus.cc:385] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.166265 7459 raft_consensus.cc:740] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.143325 7477 ts_tablet_manager.cc:1916] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.157765 7467 raft_consensus.cc:740] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.170181 7467 consensus_queue.cc:260] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.175361 7467 ts_tablet_manager.cc:1434] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.051s user 0.007s sys 0.000s
I20260317 21:18:56.178089 7459 consensus_queue.cc:260] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.182351 7467 tablet_bootstrap.cc:492] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:56.188081 7467 tablet_bootstrap.cc:654] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.191140 6673 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 2b287c83b77548f19546927b05905746
I20260317 21:18:56.193109 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.194885 7459 ts_tablet_manager.cc:1434] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.047s user 0.001s sys 0.004s
I20260317 21:18:56.195057 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.197014 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.213375 7459 tablet_bootstrap.cc:492] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:56.218190 7549 raft_consensus.cc:493] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.218755 7549 raft_consensus.cc:515] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.208292 6670 catalog_manager.cc:5671] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:56.220955 7549 leader_election.cc:290] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:56.220213 6670 catalog_manager.cc:5671] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:56.222205 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:56.224006 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:56.225668 7483 ts_tablet_manager.cc:1929] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260317 21:18:56.225922 7261 leader_election.cc:343] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.226243 7483 log.cc:1199] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/56495c7cf3964743b96ab885afdaa152
W20260317 21:18:56.226912 7264 leader_election.cc:343] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.227306 7264 leader_election.cc:304] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.227674 7483 ts_tablet_manager.cc:1950] T 56495c7cf3964743b96ab885afdaa152 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:56.228356 7549 raft_consensus.cc:2749] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.222577 6670 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 06877bd7155842b9a5ae4a23ea68ada0 (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 2 got 0
I20260317 21:18:56.229771 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 84) took 1169 ms. Trace:
I20260317 21:18:56.230137 7483 rpcz_store.cc:276] 0317 21:18:55.060426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.060565 (+ 139us) service_pool.cc:225] Handling call
0317 21:18:56.026637 (+966072us) ts_tablet_manager.cc:1151] Deleting tablet 56495c7cf3964743b96ab885afdaa152
0317 21:18:56.054388 (+ 27751us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.165204 (+110816us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.225290 (+ 60086us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.229708 (+ 4418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":965420}
I20260317 21:18:56.231513 7483 tablet_replica.cc:333] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:56.232394 7483 raft_consensus.cc:2243] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.235165 7467 tablet_bootstrap.cc:492] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:56.235678 7467 ts_tablet_manager.cc:1403] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.054s user 0.002s sys 0.008s
I20260317 21:18:56.237490 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet b196f1e1572f4a62b45df7ffc581cd7a with delete_type TABLET_DATA_DELETED (replaced by tablet a978fa28a31b4d589031c77526b62b8e) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.238761 7467 raft_consensus.cc:359] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.239394 7467 raft_consensus.cc:385] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.239634 7467 raft_consensus.cc:740] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.240250 7467 consensus_queue.cc:260] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.241325 7483 raft_consensus.cc:2272] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.248706 7459 tablet_bootstrap.cc:654] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.250025 7483 ts_tablet_manager.cc:1916] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.254809 7467 ts_tablet_manager.cc:1434] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260317 21:18:56.267391 7467 tablet_bootstrap.cc:492] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:56.268975 7477 ts_tablet_manager.cc:1929] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:56.270265 7477 log.cc:1199] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/291e2bee709243f09b338d99edbe0cb1
I20260317 21:18:56.271649 7477 ts_tablet_manager.cc:1950] T 291e2bee709243f09b338d99edbe0cb1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:56.273950 7467 tablet_bootstrap.cc:654] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.275305 7462 tablet_bootstrap.cc:654] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.277289 7459 tablet_bootstrap.cc:492] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:56.277791 7459 ts_tablet_manager.cc:1403] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.002s
I20260317 21:18:56.229156 6934 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 2b287c83b77548f19546927b05905746
I20260317 21:18:56.231671 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 56495c7cf3964743b96ab885afdaa152 (table default.Table_b1d7ee6663fe432a8940df97de65d99b [id=251402e78beb409192f3bd43ff21a9e6]) successfully deleted
I20260317 21:18:56.246243 6674 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 82f8dd260a134810a50465e4a77848ca
W20260317 21:18:56.281042 6661 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 49383.7us
I20260317 21:18:56.282013 6674 catalog_manager.cc:6741] ReplaceTablet: tablet 82f8dd260a134810a50465e4a77848ca deleted and replaced by tablet 0fd3e35e25bf444cbd821ebd9f5ee018
I20260317 21:18:56.283797 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 82f8dd260a134810a50465e4a77848ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd3e35e25bf444cbd821ebd9f5ee018) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.285459 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 82f8dd260a134810a50465e4a77848ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd3e35e25bf444cbd821ebd9f5ee018) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.255772 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:18:56.286082 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 82f8dd260a134810a50465e4a77848ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd3e35e25bf444cbd821ebd9f5ee018) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.287550 6667 catalog_manager.cc:6741] ReplaceTablet: tablet 8ba0874a8cd746e9a33596ea252b3b14 deleted and replaced by tablet 323717acdee841aaa1795a3ca851b40f
I20260317 21:18:56.288410 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 with delete_type TABLET_DATA_DELETED (replaced by tablet 323717acdee841aaa1795a3ca851b40f) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.291528 7459 raft_consensus.cc:359] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.292148 7459 raft_consensus.cc:385] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.295723 7467 tablet_bootstrap.cc:492] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:56.257666 6668 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet e1094068903840a8ab5d07c7fb0635bf (table default.Table_b8fd10368ce242839da23252952d7600 [id=93623d5786f645c7a1ad496e656caa24]). Expected version 1 got 0
I20260317 21:18:56.296434 7580 raft_consensus.cc:493] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.296887 7580 raft_consensus.cc:515] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.298699 7580 leader_election.cc:290] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:56.300031 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f749358db0f74ef78d9eee23b72bfe3e consensus state has no leader
I20260317 21:18:56.296175 7467 ts_tablet_manager.cc:1403] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.029s user 0.010s sys 0.000s
I20260317 21:18:56.300290 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:56.301003 7329 raft_consensus.cc:2468] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:56.303135 7119 leader_election.cc:304] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:56.303015 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:56.292373 7459 raft_consensus.cc:740] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.304427 7580 raft_consensus.cc:2804] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:56.305023 7549 raft_consensus.cc:493] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.305353 7580 raft_consensus.cc:493] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:56.305657 7580 raft_consensus.cc:3060] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:56.305220 7459 consensus_queue.cc:260] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.305462 7549 raft_consensus.cc:515] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.305893 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 with delete_type TABLET_DATA_DELETED (replaced by tablet 323717acdee841aaa1795a3ca851b40f) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.306584 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 with delete_type TABLET_DATA_DELETED (replaced by tablet 323717acdee841aaa1795a3ca851b40f) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.307430 7549 leader_election.cc:290] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:56.307731 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 291e2bee709243f09b338d99edbe0cb1 (table default.Table_2fab17210bfa496ba34bb9580bee9d79 [id=040d2e5e61ef41e4ac8953a3d63a226c]) successfully deleted
I20260317 21:18:56.308797 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_ce47367ed77d4c06ad8c624c31dd3b2b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:56.309126 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "589aadc7897b422e90d3f7caed76ab59" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:56.306789 7477 tablet_replica.cc:333] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:18:56.311475 7120 leader_election.cc:343] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:56.311707 7261 leader_election.cc:343] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:56.311038 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce47367ed77d4c06ad8c624c31dd3b2b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.313647 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "589aadc7897b422e90d3f7caed76ab59" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:56.313560 7467 raft_consensus.cc:359] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.314109 7477 raft_consensus.cc:2243] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.314226 7467 raft_consensus.cc:385] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.314448 7467 raft_consensus.cc:740] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.314586 7477 raft_consensus.cc:2272] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:18:56.316151 7264 leader_election.cc:343] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.316550 7264 leader_election.cc:304] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.317246 7483 ts_tablet_manager.cc:1929] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:56.317586 7483 log.cc:1199] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/be2c469c792047d88fcf2ff9d53b7832
I20260317 21:18:56.318030 7477 ts_tablet_manager.cc:1916] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.319306 7580 raft_consensus.cc:515] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.318617 7549 raft_consensus.cc:2749] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.319917 7308 tablet_service.cc:1558] Processing DeleteTablet for tablet 60f3d32e1f7649fa92861f51134cf000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.319096 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.321035 7483 ts_tablet_manager.cc:1950] T be2c469c792047d88fcf2ff9d53b7832 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:56.322255 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.322973 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 85) took 1260 ms. Trace:
I20260317 21:18:56.323299 7483 rpcz_store.cc:276] 0317 21:18:55.062103 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.064752 (+ 2649us) service_pool.cc:225] Handling call
0317 21:18:56.231263 (+1166511us) ts_tablet_manager.cc:1151] Deleting tablet be2c469c792047d88fcf2ff9d53b7832
0317 21:18:56.231351 (+ 88us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.262389 (+ 31038us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.317025 (+ 54636us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.322914 (+ 5889us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1165877}
I20260317 21:18:56.324438 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.325017 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet be2c469c792047d88fcf2ff9d53b7832 (table default.Table_ffede0d29ebf494f8eec15158eeea6f6 [id=e9385f5ddf6e430a83595a74932a704e]) successfully deleted
I20260317 21:18:56.315045 7467 consensus_queue.cc:260] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.327726 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:56.328842 7459 ts_tablet_manager.cc:1434] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.051s user 0.006s sys 0.001s
W20260317 21:18:56.330574 7120 leader_election.cc:343] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.334300 7459 tablet_bootstrap.cc:492] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:56.334195 6669 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet d59deea79cfb4a83831bb0227797c690
I20260317 21:18:56.336068 7549 raft_consensus.cc:493] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.336562 7549 raft_consensus.cc:515] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.337035 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet d59deea79cfb4a83831bb0227797c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.339485 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60f3d32e1f7649fa92861f51134cf000" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:56.340827 7264 leader_election.cc:343] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.340583 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_efd3fbe665cd4ca4ab6be9a618c17a6b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:56.342823 7459 tablet_bootstrap.cc:654] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.342788 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60f3d32e1f7649fa92861f51134cf000" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:56.343163 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet d59deea79cfb4a83831bb0227797c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:56.344688 7261 leader_election.cc:343] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.344404 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e97bdfbd243044eca8ec69071ca2cbae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:56.345095 7261 leader_election.cc:304] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.345714 7580 leader_election.cc:290] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.346174 7467 ts_tablet_manager.cc:1434] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.046s user 0.007s sys 0.000s
I20260317 21:18:56.346024 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
W20260317 21:18:56.346232 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e97bdfbd243044eca8ec69071ca2cbae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.346556 7329 raft_consensus.cc:3060] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:56.347002 7656 raft_consensus.cc:2749] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.347394 7467 tablet_bootstrap.cc:492] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:56.324463 7483 tablet_replica.cc:333] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:56.350590 7549 leader_election.cc:290] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:18:56.357641 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fd3e35e25bf444cbd821ebd9f5ee018 has no consensus state
I20260317 21:18:56.359385 7329 raft_consensus.cc:2468] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:56.350796 7483 raft_consensus.cc:2243] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.360580 7119 leader_election.cc:304] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.361662 7580 raft_consensus.cc:2804] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:56.362776 7580 raft_consensus.cc:697] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:56.365152 7580 consensus_queue.cc:237] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.366276 7483 raft_consensus.cc:2272] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:18:56.366446 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_efd3fbe665cd4ca4ab6be9a618c17a6b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.370500 7483 ts_tablet_manager.cc:1916] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.374686 7467 tablet_bootstrap.cc:654] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.374752 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.383471 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet d59deea79cfb4a83831bb0227797c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.388149 7010 tablet_service.cc:1511] Processing CreateTablet for tablet d05973b63a8c445995ca51d8aec7bcf9 (DEFAULT_TABLE table=default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.389627 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05973b63a8c445995ca51d8aec7bcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.395736 7459 tablet_bootstrap.cc:492] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:56.396193 7459 ts_tablet_manager.cc:1403] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.001s
I20260317 21:18:56.398429 7467 tablet_bootstrap.cc:492] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:56.398979 7467 ts_tablet_manager.cc:1403] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.052s user 0.012s sys 0.000s
I20260317 21:18:56.401274 7467 raft_consensus.cc:359] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.401262 7459 raft_consensus.cc:359] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.401953 7459 raft_consensus.cc:385] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.402019 7467 raft_consensus.cc:385] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.402242 7459 raft_consensus.cc:740] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.402271 7467 raft_consensus.cc:740] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.406951 7159 tablet_service.cc:1511] Processing CreateTablet for tablet d05973b63a8c445995ca51d8aec7bcf9 (DEFAULT_TABLE table=default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.407487 7477 ts_tablet_manager.cc:1929] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:56.407841 7477 log.cc:1199] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/be2c469c792047d88fcf2ff9d53b7832
I20260317 21:18:56.408066 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05973b63a8c445995ca51d8aec7bcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.408514 7477 ts_tablet_manager.cc:1950] T be2c469c792047d88fcf2ff9d53b7832 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:56.410459 7483 ts_tablet_manager.cc:1929] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:56.410842 7483 log.cc:1199] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/99b954ac41134300823458a7a25ef738
I20260317 21:18:56.411080 7477 tablet_replica.cc:333] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:56.412317 7467 consensus_queue.cc:260] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.415395 7467 ts_tablet_manager.cc:1434] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.016s user 0.008s sys 0.000s
I20260317 21:18:56.416360 7467 tablet_bootstrap.cc:492] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:18:56.385264 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f [id=bcbc5e44c4ce46d2b795f6997fa466ce], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 0fd3e35e25bf444cbd821ebd9f5ee018 has no consensus state
I20260317 21:18:56.414462 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet be2c469c792047d88fcf2ff9d53b7832 (table default.Table_ffede0d29ebf494f8eec15158eeea6f6 [id=e9385f5ddf6e430a83595a74932a704e]) successfully deleted
W20260317 21:18:56.420415 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 2b287c83b77548f19546927b05905746 because the tablet was not found. No further retry: Not found: Tablet not found: 2b287c83b77548f19546927b05905746
I20260317 21:18:56.422163 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.411614 7483 ts_tablet_manager.cc:1950] T 99b954ac41134300823458a7a25ef738 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:56.429587 7580 raft_consensus.cc:493] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.430222 7580 raft_consensus.cc:515] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.433279 7580 leader_election.cc:290] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.428350 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 87) took 1174 ms. Trace:
I20260317 21:18:56.439476 7467 tablet_bootstrap.cc:654] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.433772 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1094068903840a8ab5d07c7fb0635bf" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:56.440122 7329 raft_consensus.cc:2468] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:56.413604 7459 consensus_queue.cc:260] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.439903 7483 rpcz_store.cc:276] 0317 21:18:55.253464 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.253565 (+ 101us) service_pool.cc:225] Handling call
0317 21:18:56.324267 (+1070702us) ts_tablet_manager.cc:1151] Deleting tablet 99b954ac41134300823458a7a25ef738
0317 21:18:56.324336 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.392255 (+ 67919us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.410256 (+ 18001us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.428287 (+ 18031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070072}
I20260317 21:18:56.441249 7483 tablet_replica.cc:333] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:56.433728 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1094068903840a8ab5d07c7fb0635bf" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:56.441668 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 99b954ac41134300823458a7a25ef738 (table default.Table_93723f1459114d9ba82a21cee9158b3d [id=d079fd8ed7aa48ad9420c7cf7419c36e]) successfully deleted
I20260317 21:18:56.442400 7119 leader_election.cc:304] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:56.442641 7304 tablet_service.cc:1511] Processing CreateTablet for tablet d05973b63a8c445995ca51d8aec7bcf9 (DEFAULT_TABLE table=default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.443616 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d05973b63a8c445995ca51d8aec7bcf9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.444949 7012 tablet_service.cc:1511] Processing CreateTablet for tablet eb332f981fc84a178fadae2f4908bdde (DEFAULT_TABLE table=default.Table_0c38032670874e6f85493a54b207749e [id=10eb9e3956be42aab8e5bfc43d9b9e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.447518 7459 ts_tablet_manager.cc:1434] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.051s user 0.006s sys 0.001s
W20260317 21:18:56.448714 7120 leader_election.cc:343] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.449018 7154 tablet_service.cc:1511] Processing CreateTablet for tablet eb332f981fc84a178fadae2f4908bdde (DEFAULT_TABLE table=default.Table_0c38032670874e6f85493a54b207749e [id=10eb9e3956be42aab8e5bfc43d9b9e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.450045 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb332f981fc84a178fadae2f4908bdde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.450311 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb332f981fc84a178fadae2f4908bdde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.452005 7580 raft_consensus.cc:2804] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:56.452353 7580 raft_consensus.cc:493] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:56.452746 7580 raft_consensus.cc:3060] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:56.441953 7483 raft_consensus.cc:2243] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.468832 7459 tablet_bootstrap.cc:492] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:56.473233 7477 raft_consensus.cc:2243] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.473925 7477 raft_consensus.cc:2272] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.475833 7483 raft_consensus.cc:2272] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.481350 7483 ts_tablet_manager.cc:1916] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.478258 7477 ts_tablet_manager.cc:1916] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.489892 7459 tablet_bootstrap.cc:654] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.502257 6668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0a38822238d94d4eaadded2ca39840f4" } modify_external_catalogs: true
I20260317 21:18:56.502887 6668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0a38822238d94d4eaadded2ca39840f4" } modify_external_catalogs: true
I20260317 21:18:56.488196 7462 tablet_bootstrap.cc:492] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:56.512419 7462 ts_tablet_manager.cc:1403] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.392s user 0.012s sys 0.000s
I20260317 21:18:56.518232 7462 raft_consensus.cc:359] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.523445 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet c1e672ca556e400ab05763abcd49ad77 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:56.524477 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ae0991c6e7cf4fc1918875b5b48c1062" } new_table_name: "default.Table_6d4e07837b8945f981be0ae07f5b4ea0" modify_external_catalogs: true
I20260317 21:18:56.534240 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_87cbb52aa7e249f98d6eb6ed317d3d46"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:56.536589 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87cbb52aa7e249f98d6eb6ed317d3d46 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.519265 7462 raft_consensus.cc:385] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.537055 7462 raft_consensus.cc:740] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.537948 7462 consensus_queue.cc:260] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.541211 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet 935faf7f9129440a91635bfa96191cd6 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:56.542512 7459 tablet_bootstrap.cc:492] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:56.543052 7459 ts_tablet_manager.cc:1403] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.000s
I20260317 21:18:56.546784 7459 raft_consensus.cc:359] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.547415 7459 raft_consensus.cc:385] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.547669 7459 raft_consensus.cc:740] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.562340 7459 consensus_queue.cc:260] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.563758 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f701f4a89ae64b4db661f19343773e30" } new_table_name: "default.Table_1e2eb35c075a45d1afe94750072f9341" modify_external_catalogs: true
I20260317 21:18:56.565591 7462 ts_tablet_manager.cc:1434] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.053s user 0.008s sys 0.000s
I20260317 21:18:56.566658 7462 tablet_bootstrap.cc:492] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:56.571524 7459 ts_tablet_manager.cc:1434] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.027s user 0.005s sys 0.000s
I20260317 21:18:56.575054 7580 raft_consensus.cc:515] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.582159 7459 tablet_bootstrap.cc:492] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:56.592826 7304 tablet_service.cc:1511] Processing CreateTablet for tablet eb332f981fc84a178fadae2f4908bdde (DEFAULT_TABLE table=default.Table_0c38032670874e6f85493a54b207749e [id=10eb9e3956be42aab8e5bfc43d9b9e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.593264 7477 ts_tablet_manager.cc:1929] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:56.593636 7477 log.cc:1199] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/b196f1e1572f4a62b45df7ffc581cd7a
I20260317 21:18:56.594175 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb332f981fc84a178fadae2f4908bdde. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.594339 7477 ts_tablet_manager.cc:1950] T b196f1e1572f4a62b45df7ffc581cd7a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:56.597051 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 0eee8b5588c24661889c593797c94269 (DEFAULT_TABLE table=default.Table_212ed8405de344519eaecc32555fcbb5 [id=8fee88b3acf649ffbfbc18d16f47af40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.598299 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eee8b5588c24661889c593797c94269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.604223 7462 tablet_bootstrap.cc:654] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.580787 7467 tablet_bootstrap.cc:492] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:56.609028 7467 ts_tablet_manager.cc:1403] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.193s user 0.014s sys 0.000s
I20260317 21:18:56.610927 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet b196f1e1572f4a62b45df7ffc581cd7a (table default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3]) successfully deleted
I20260317 21:18:56.611447 7467 raft_consensus.cc:359] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.612581 7467 raft_consensus.cc:385] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.613050 7467 raft_consensus.cc:740] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.614145 7467 consensus_queue.cc:260] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:56.616861 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 82f8dd260a134810a50465e4a77848ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82f8dd260a134810a50465e4a77848ca already in progress: creating tablet
W20260317 21:18:56.622720 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 8ba0874a8cd746e9a33596ea252b3b14 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ba0874a8cd746e9a33596ea252b3b14 already in progress: creating tablet
I20260317 21:18:56.623324 7580 leader_election.cc:290] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.623539 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1094068903840a8ab5d07c7fb0635bf" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:56.621948 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e1094068903840a8ab5d07c7fb0635bf" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:56.623674 7467 ts_tablet_manager.cc:1434] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
I20260317 21:18:56.624114 7329 raft_consensus.cc:3060] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:56.625144 7467 tablet_bootstrap.cc:492] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:18:56.626114 7120 leader_election.cc:343] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.614763 7459 tablet_bootstrap.cc:654] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.628399 7549 raft_consensus.cc:493] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:18:56.628901 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 2b287c83b77548f19546927b05905746 because the tablet was not found. No further retry: Not found: Tablet not found: 2b287c83b77548f19546927b05905746
I20260317 21:18:56.628857 7549 raft_consensus.cc:515] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.630616 7549 leader_election.cc:290] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:18:56.630596 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet d59deea79cfb4a83831bb0227797c690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d59deea79cfb4a83831bb0227797c690 already in progress: creating tablet
I20260317 21:18:56.632889 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:56.634198 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:56.635511 7261 leader_election.cc:343] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:56.639459 7264 leader_election.cc:343] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.639868 7264 leader_election.cc:304] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.640973 7483 ts_tablet_manager.cc:1929] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:56.641044 7478 ts_tablet_manager.cc:1929] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:56.640942 7549 raft_consensus.cc:2749] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.641394 7478 log.cc:1199] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/b196f1e1572f4a62b45df7ffc581cd7a
I20260317 21:18:56.641405 7483 log.cc:1199] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:18:56.642089 7483 ts_tablet_manager.cc:1950] T b6fb7cb082cf47ce8cf2a98146633af7 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:56.644299 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 90) took 1029 ms. Trace:
I20260317 21:18:56.648947 7478 ts_tablet_manager.cc:1950] T b196f1e1572f4a62b45df7ffc581cd7a P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:56.651408 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 72) took 2176 ms. Trace:
I20260317 21:18:56.651870 7478 rpcz_store.cc:276] 0317 21:18:54.475105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.477471 (+ 2366us) service_pool.cc:225] Handling call
0317 21:18:56.005764 (+1528293us) ts_tablet_manager.cc:1151] Deleting tablet b196f1e1572f4a62b45df7ffc581cd7a
0317 21:18:56.005808 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.234591 (+228783us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.640799 (+406208us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.651349 (+ 10550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527613}
I20260317 21:18:56.652818 7483 rpcz_store.cc:276] 0317 21:18:55.615351 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.620771 (+ 5420us) service_pool.cc:225] Handling call
0317 21:18:56.441087 (+820316us) ts_tablet_manager.cc:1151] Deleting tablet b6fb7cb082cf47ce8cf2a98146633af7
0317 21:18:56.441133 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.577149 (+136016us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.640744 (+ 63595us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.644186 (+ 3442us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":819605}
I20260317 21:18:56.645596 7329 raft_consensus.cc:2468] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:56.657471 7459 tablet_bootstrap.cc:492] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:56.657917 7459 ts_tablet_manager.cc:1403] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.076s user 0.010s sys 0.002s
I20260317 21:18:56.662225 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 0eee8b5588c24661889c593797c94269 (DEFAULT_TABLE table=default.Table_212ed8405de344519eaecc32555fcbb5 [id=8fee88b3acf649ffbfbc18d16f47af40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.662884 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 35023195b9094d58827921a1dcb2ff05 (DEFAULT_TABLE table=default.Table_02290a1a580945fc935a328e9a9faa11 [id=99a93aed10d2424087ea303d139653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.663125 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet b6fb7cb082cf47ce8cf2a98146633af7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]) successfully deleted
I20260317 21:18:56.663329 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eee8b5588c24661889c593797c94269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.664093 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35023195b9094d58827921a1dcb2ff05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.664238 7483 tablet_replica.cc:333] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:56.672792 7483 raft_consensus.cc:2243] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:56.675383 7462 tablet_bootstrap.cc:492] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:56.675853 7462 ts_tablet_manager.cc:1403] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.109s user 0.011s sys 0.001s
I20260317 21:18:56.677080 7459 raft_consensus.cc:359] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.680800 7462 raft_consensus.cc:359] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.681651 7462 raft_consensus.cc:385] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.651589 7467 tablet_bootstrap.cc:654] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.654961 7119 leader_election.cc:304] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.662045 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 0eee8b5588c24661889c593797c94269 (DEFAULT_TABLE table=default.Table_212ed8405de344519eaecc32555fcbb5 [id=8fee88b3acf649ffbfbc18d16f47af40]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.682489 7580 raft_consensus.cc:2804] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:56.683036 7580 raft_consensus.cc:697] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:56.683913 7580 consensus_queue.cc:237] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.691105 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0eee8b5588c24661889c593797c94269. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.693194 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet b196f1e1572f4a62b45df7ffc581cd7a (table default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3]) successfully deleted
I20260317 21:18:56.693660 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 74) took 2151 ms. Trace:
I20260317 21:18:56.694074 7478 rpcz_store.cc:276] 0317 21:18:54.542870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.543082 (+ 212us) service_pool.cc:225] Handling call
0317 21:18:56.693326 (+2150244us) ts_tablet_manager.cc:1151] Deleting tablet c6d5385171364a078d968bb0a489383a
0317 21:18:56.693369 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.693607 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2149479}
I20260317 21:18:56.695142 7483 raft_consensus.cc:2272] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:56.695849 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 35023195b9094d58827921a1dcb2ff05 (DEFAULT_TABLE table=default.Table_02290a1a580945fc935a328e9a9faa11 [id=99a93aed10d2424087ea303d139653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.697460 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35023195b9094d58827921a1dcb2ff05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.699432 7459 raft_consensus.cc:385] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.699659 7483 ts_tablet_manager.cc:1916] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:56.699729 7459 raft_consensus.cc:740] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.700364 7459 consensus_queue.cc:260] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.681972 7462 raft_consensus.cc:740] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.705194 7672 raft_consensus.cc:493] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.705687 7672 raft_consensus.cc:515] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.706102 7462 consensus_queue.cc:260] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.707516 7672 leader_election.cc:290] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.709890 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99b954ac41134300823458a7a25ef738" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:56.710089 7119 leader_election.cc:343] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 99b954ac41134300823458a7a25ef738
I20260317 21:18:56.710843 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 35023195b9094d58827921a1dcb2ff05 (DEFAULT_TABLE table=default.Table_02290a1a580945fc935a328e9a9faa11 [id=99a93aed10d2424087ea303d139653c2]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:56.711408 7120 leader_election.cc:343] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.711794 7120 leader_election.cc:304] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.713724 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 564a02888d854ea58d7a590cba816b86 (DEFAULT_TABLE table=default.Table_b451641b2795486b8d68f72ab543b5b2 [id=22768a64f9394820997448f0c81605ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.714802 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564a02888d854ea58d7a590cba816b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.717085 7672 raft_consensus.cc:2749] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.717209 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 564a02888d854ea58d7a590cba816b86 (DEFAULT_TABLE table=default.Table_b451641b2795486b8d68f72ab543b5b2 [id=22768a64f9394820997448f0c81605ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.718355 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564a02888d854ea58d7a590cba816b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.721161 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 35023195b9094d58827921a1dcb2ff05. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.723825 7459 ts_tablet_manager.cc:1434] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.066s user 0.007s sys 0.000s
I20260317 21:18:56.727296 7549 raft_consensus.cc:493] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.727753 7549 raft_consensus.cc:515] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.730114 7549 leader_election.cc:290] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:56.731271 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:56.733050 7261 leader_election.cc:343] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.733999 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:56.735334 7264 leader_election.cc:343] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.735739 7264 leader_election.cc:304] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:56.736117 7459 tablet_bootstrap.cc:492] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:56.736725 7549 raft_consensus.cc:2749] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:56.745951 7467 tablet_bootstrap.cc:492] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:56.746390 7467 ts_tablet_manager.cc:1403] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.121s user 0.004s sys 0.009s
I20260317 21:18:56.748870 6668 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 5a81b3ced72e473292a29489184ad55c
I20260317 21:18:56.758277 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.759610 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 56495c7cf3964743b96ab885afdaa152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.763610 7528 raft_consensus.cc:493] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.764093 7528 raft_consensus.cc:515] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.767565 7528 leader_election.cc:290] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.768682 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56495c7cf3964743b96ab885afdaa152" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:56.769338 7184 raft_consensus.cc:2468] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:56.769577 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:56.771656 6971 leader_election.cc:304] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:18:56.773720 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet c6d5385171364a078d968bb0a489383a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c6d5385171364a078d968bb0a489383a already in progress: creating tablet
I20260317 21:18:56.777053 7528 raft_consensus.cc:2804] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:56.777488 7528 raft_consensus.cc:493] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:56.777844 7528 raft_consensus.cc:3060] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:56.789350 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 564a02888d854ea58d7a590cba816b86 (DEFAULT_TABLE table=default.Table_b451641b2795486b8d68f72ab543b5b2 [id=22768a64f9394820997448f0c81605ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.790139 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 2b287c83b77548f19546927b05905746 (DEFAULT_TABLE table=default.Table_e4715bec3faa471cbdcb63c57766180a [id=ff96f5df47e0414290054b29ed53479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.793485 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b287c83b77548f19546927b05905746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.793646 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 76) took 2192 ms. Trace:
I20260317 21:18:56.795262 7478 rpcz_store.cc:276] 0317 21:18:54.601019 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.601185 (+ 166us) service_pool.cc:225] Handling call
0317 21:18:56.784984 (+2183799us) ts_tablet_manager.cc:1151] Deleting tablet 163cc8c04ca94a269c7ae2efaa162525
0317 21:18:56.791902 (+ 6918us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.793567 (+ 1665us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2182771}
I20260317 21:18:56.733104 7462 ts_tablet_manager.cc:1434] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.057s user 0.009s sys 0.000s
I20260317 21:18:56.798162 7459 tablet_bootstrap.cc:654] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.806700 7672 raft_consensus.cc:493] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:56.807461 7672 raft_consensus.cc:515] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.811543 7528 raft_consensus.cc:515] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.812945 7672 leader_election.cc:290] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:56.808104 7478 tablet_replica.cc:333] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:56.814370 7478 raft_consensus.cc:2243] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:56.815737 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:56.816355 7327 raft_consensus.cc:2468] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:56.818068 7119 leader_election.cc:304] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:56.819770 7672 raft_consensus.cc:2804] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:56.820158 7672 raft_consensus.cc:493] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:56.812212 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:56.817143 6973 leader_election.cc:343] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 56495c7cf3964743b96ab885afdaa152
I20260317 21:18:56.820614 7580 consensus_queue.cc:1048] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:18:56.820766 7672 raft_consensus.cc:3060] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:56.822285 7120 leader_election.cc:343] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:56.805811 7467 raft_consensus.cc:359] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.825598 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "56495c7cf3964743b96ab885afdaa152" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:56.825943 7467 raft_consensus.cc:385] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.826195 7184 raft_consensus.cc:3060] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:56.827811 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 163cc8c04ca94a269c7ae2efaa162525 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 163cc8c04ca94a269c7ae2efaa162525 already in progress: creating tablet
I20260317 21:18:56.829414 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564a02888d854ea58d7a590cba816b86. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.830140 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 2b287c83b77548f19546927b05905746 (DEFAULT_TABLE table=default.Table_e4715bec3faa471cbdcb63c57766180a [id=ff96f5df47e0414290054b29ed53479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.831314 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b287c83b77548f19546927b05905746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.833357 7528 leader_election.cc:290] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:56.833928 6973 leader_election.cc:343] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 56495c7cf3964743b96ab885afdaa152
I20260317 21:18:56.835651 7459 tablet_bootstrap.cc:492] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:56.836112 7459 ts_tablet_manager.cc:1403] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.100s user 0.010s sys 0.001s
I20260317 21:18:56.839560 7459 raft_consensus.cc:359] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.841450 7462 tablet_bootstrap.cc:492] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:56.845278 7459 raft_consensus.cc:385] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:56.845686 7459 raft_consensus.cc:740] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:56.859113 7459 consensus_queue.cc:260] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.863600 7483 ts_tablet_manager.cc:1929] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:56.867002 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 78e2e20d320547b3bb1e03923de52822 (DEFAULT_TABLE table=default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.868348 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e2e20d320547b3bb1e03923de52822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.869722 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_c846e042b2d4448ab3aa5e50bda65ec4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:56.870816 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 78e2e20d320547b3bb1e03923de52822 (DEFAULT_TABLE table=default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:56.871249 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c846e042b2d4448ab3aa5e50bda65ec4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.872123 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e2e20d320547b3bb1e03923de52822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.873356 7459 ts_tablet_manager.cc:1434] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.037s user 0.002s sys 0.005s
I20260317 21:18:56.877684 6673 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_222217e8357d431baa8615150079a999" } modify_external_catalogs: true
I20260317 21:18:56.878378 6673 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_222217e8357d431baa8615150079a999" } modify_external_catalogs: true
I20260317 21:18:56.881341 7459 tablet_bootstrap.cc:492] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:56.880904 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_c1eb7feff9ea4b568cb56bb3761a2349"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:56.883028 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1eb7feff9ea4b568cb56bb3761a2349 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.867058 7308 tablet_service.cc:1511] Processing CreateTablet for tablet 2b287c83b77548f19546927b05905746 (DEFAULT_TABLE table=default.Table_e4715bec3faa471cbdcb63c57766180a [id=ff96f5df47e0414290054b29ed53479a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.869580 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 78e2e20d320547b3bb1e03923de52822 (DEFAULT_TABLE table=default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.887405 7483 log.cc:1199] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/27977b00ae1a4e7ca2eddaa693d11a65
I20260317 21:18:56.888170 6668 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 06877bd7155842b9a5ae4a23ea68ada0 (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 2 got 0
I20260317 21:18:56.901472 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (DEFAULT_TABLE table=default.Table_638c2b3200334f309baef743312b646c [id=81dc7f934d2346cd8ad83e5d718c5d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.902880 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb432b6bccf4e32a2fa9d4979586c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.917567 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (DEFAULT_TABLE table=default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.918840 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6275af1c9c41ae8e4515deabdbdee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.930306 7459 tablet_bootstrap.cc:654] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:56.944190 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (DEFAULT_TABLE table=default.Table_f701f4a89ae64b4db661f19343773e30 [id=f652dc1222a3411db5a54ad8e8927a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.945493 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6caf05c79a5f470e8b7796d0e85e36f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.950568 7184 raft_consensus.cc:2468] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:56.952234 6971 leader_election.cc:304] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:56.953040 7528 raft_consensus.cc:2804] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
W20260317 21:18:56.956584 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59fce28afbeb4d8eadf63f9a97d1183d consensus state has no leader
I20260317 21:18:56.944221 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b6fea0d08b2b48609aed18c1983a38a7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:56.959800 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b6fea0d08b2b48609aed18c1983a38a7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:56.911520 6669 catalog_manager.cc:5671] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:56.964151 6669 catalog_manager.cc:5671] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:56.979305 7528 raft_consensus.cc:697] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:56.980688 7528 consensus_queue.cc:237] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:56.985406 7158 tablet_service.cc:1511] Processing CreateTablet for tablet a978fa28a31b4d589031c77526b62b8e (DEFAULT_TABLE table=default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:56.987244 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a978fa28a31b4d589031c77526b62b8e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:56.971030 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6caf05c79a5f470e8b7796d0e85e36f1 consensus state has no leader
I20260317 21:18:56.988565 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet 60f3d32e1f7649fa92861f51134cf000 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:56.974963 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 935faf7f9129440a91635bfa96191cd6
I20260317 21:18:56.990738 6667 catalog_manager.cc:6741] ReplaceTablet: tablet 935faf7f9129440a91635bfa96191cd6 deleted and replaced by tablet 57e721ac4b24438b9b2a0e2569027236
I20260317 21:18:56.980329 6674 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e97bdfbd243044eca8ec69071ca2cbae" } modify_external_catalogs: true
I20260317 21:18:56.992273 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 935faf7f9129440a91635bfa96191cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet 57e721ac4b24438b9b2a0e2569027236) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:56.992343 6674 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e97bdfbd243044eca8ec69071ca2cbae" } modify_external_catalogs: true
I20260317 21:18:57.002533 7459 tablet_bootstrap.cc:492] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:57.003105 7459 ts_tablet_manager.cc:1403] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.122s user 0.006s sys 0.005s
W20260317 21:18:56.983232 6662 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 115850us
I20260317 21:18:57.004846 7036 raft_consensus.cc:3060] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.005573 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d (DEFAULT_TABLE table=default.Table_6d4e07837b8945f981be0ae07f5b4ea0 [id=2e0f6bf62c664a6cb3f2bdad9a74e11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.006734 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fce28afbeb4d8eadf63f9a97d1183d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.008188 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (DEFAULT_TABLE table=default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.008782 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 56495c7cf3964743b96ab885afdaa152 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:57.009641 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6275af1c9c41ae8e4515deabdbdee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.009691 7017 tablet_service.cc:1511] Processing CreateTablet for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (DEFAULT_TABLE table=default.Table_f701f4a89ae64b4db661f19343773e30 [id=f652dc1222a3411db5a54ad8e8927a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.010457 7008 tablet_service.cc:1511] Processing CreateTablet for tablet a978fa28a31b4d589031c77526b62b8e (DEFAULT_TABLE table=default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.010846 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6caf05c79a5f470e8b7796d0e85e36f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.011520 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a978fa28a31b4d589031c77526b62b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.011974 7477 tablet_replica.cc:333] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:57.007352 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (DEFAULT_TABLE table=default.Table_638c2b3200334f309baef743312b646c [id=81dc7f934d2346cd8ad83e5d718c5d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.014701 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb432b6bccf4e32a2fa9d4979586c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:56.826208 7467 raft_consensus.cc:740] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.022099 6669 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet e1094068903840a8ab5d07c7fb0635bf (table default.Table_b8fd10368ce242839da23252952d7600 [id=93623d5786f645c7a1ad496e656caa24]). Expected version 1 got 0
I20260317 21:18:57.025242 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet 935faf7f9129440a91635bfa96191cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet 57e721ac4b24438b9b2a0e2569027236) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:56.815017 7478 raft_consensus.cc:2272] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.028098 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d (DEFAULT_TABLE table=default.Table_6d4e07837b8945f981be0ae07f5b4ea0 [id=2e0f6bf62c664a6cb3f2bdad9a74e11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.029282 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fce28afbeb4d8eadf63f9a97d1183d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.042160 7159 tablet_service.cc:1511] Processing CreateTablet for tablet fd7cf19121b5496a849118de4a29db5b (DEFAULT_TABLE table=default.Table_713bafc801d244d1a80d9157aebd581b [id=6de336aaef3744f58b4fdfab525c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.043301 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7cf19121b5496a849118de4a29db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.020632 7467 consensus_queue.cc:260] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.048151 7467 ts_tablet_manager.cc:1434] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.302s user 0.008s sys 0.000s
I20260317 21:18:57.050834 7467 tablet_bootstrap.cc:492] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:57.054934 7014 tablet_service.cc:1511] Processing CreateTablet for tablet fd7cf19121b5496a849118de4a29db5b (DEFAULT_TABLE table=default.Table_713bafc801d244d1a80d9157aebd581b [id=6de336aaef3744f58b4fdfab525c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.055550 7478 ts_tablet_manager.cc:1916] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:57.057863 7672 raft_consensus.cc:515] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.061699 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:57.061856 7672 leader_election.cc:290] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.063045 7701 raft_consensus.cc:493] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.063891 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7cf19121b5496a849118de4a29db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.063516 7701 raft_consensus.cc:515] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.065191 7462 tablet_bootstrap.cc:700] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Time spent opening tablet: real 0.188s user 0.001s sys 0.000s
I20260317 21:18:57.066884 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (DEFAULT_TABLE table=default.Table_638c2b3200334f309baef743312b646c [id=81dc7f934d2346cd8ad83e5d718c5d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.068019 7462 tablet_bootstrap.cc:654] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.068939 7483 ts_tablet_manager.cc:1950] T 27977b00ae1a4e7ca2eddaa693d11a65 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:57.073954 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet c6d5385171364a078d968bb0a489383a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:57.077767 7120 leader_election.cc:343] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:57.077931 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 63a27911e0e545f5811eb075997d0b0f (DEFAULT_TABLE table=default.Table_ed089cf2f5554ed38d669ac4976ee78c [id=ff741e0270c34ca99f3d1e3e7c45293f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.078645 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 63a27911e0e545f5811eb075997d0b0f (DEFAULT_TABLE table=default.Table_ed089cf2f5554ed38d669ac4976ee78c [id=ff741e0270c34ca99f3d1e3e7c45293f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.079888 7477 raft_consensus.cc:2243] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:57.084703 7477 pending_rounds.cc:70] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:18:57.084971 7477 pending_rounds.cc:77] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412452285100032 op_type: NO_OP noop_request { }
I20260317 21:18:57.085345 7477 raft_consensus.cc:2889] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:57.085651 7477 raft_consensus.cc:2272] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.087241 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a27911e0e545f5811eb075997d0b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.096164 7459 raft_consensus.cc:359] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.096931 7549 raft_consensus.cc:493] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.097141 7459 raft_consensus.cc:385] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.097393 7459 raft_consensus.cc:740] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.097401 7549 raft_consensus.cc:515] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.099375 7672 consensus_queue.cc:1048] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:57.114547 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:57.101971 7459 consensus_queue.cc:260] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:57.150085 7702 consensus_queue.cc:1175] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20260317 21:18:57.150755 7459 ts_tablet_manager.cc:1434] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.147s user 0.008s sys 0.000s
I20260317 21:18:57.152539 7459 tablet_bootstrap.cc:492] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:57.117146 7701 leader_election.cc:290] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.160542 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:57.161087 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "478f55d4c3034be0b1ef336a173c55b2" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:57.161217 7327 raft_consensus.cc:3060] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.162545 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78e2e20d320547b3bb1e03923de52822. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.169360 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7eb432b6bccf4e32a2fa9d4979586c52. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.174949 7327 raft_consensus.cc:2468] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:57.185117 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6d5385171364a078d968bb0a489383a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:57.168347 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 91) took 1552 ms. Trace:
I20260317 21:18:57.187932 7483 rpcz_store.cc:276] 0317 21:18:55.615736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.621746 (+ 6010us) service_pool.cc:225] Handling call
0317 21:18:56.664089 (+1042343us) ts_tablet_manager.cc:1151] Deleting tablet 27977b00ae1a4e7ca2eddaa693d11a65
0317 21:18:56.664131 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:56.823046 (+158915us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:56.863369 (+ 40323us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.168276 (+304907us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1041637}
I20260317 21:18:57.190210 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 92) took 1574 ms. Trace:
I20260317 21:18:57.190672 7483 rpcz_store.cc:276] 0317 21:18:55.616141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.622636 (+ 6495us) service_pool.cc:225] Handling call
0317 21:18:57.189700 (+1567064us) ts_tablet_manager.cc:1151] Deleting tablet dc929bae186841d2aba674659e457301
0317 21:18:57.189773 (+ 73us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.190135 (+ 362us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566361}
I20260317 21:18:57.191870 7483 tablet_replica.cc:333] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:57.177577 7549 leader_election.cc:290] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:57.196283 7483 raft_consensus.cc:2243] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260317 21:18:57.196730 7261 leader_election.cc:343] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:57.197964 7483 raft_consensus.cc:2272] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.199007 7307 tablet_service.cc:1511] Processing CreateTablet for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (DEFAULT_TABLE table=default.Table_f701f4a89ae64b4db661f19343773e30 [id=f652dc1222a3411db5a54ad8e8927a58]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.200049 7483 ts_tablet_manager.cc:1916] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:57.199474 7309 tablet_service.cc:1511] Processing CreateTablet for tablet a978fa28a31b4d589031c77526b62b8e (DEFAULT_TABLE table=default.Table_4d068d09b4944465bfdf018355207dc3 [id=8a0a0d875f5e4f5fbcbb74938698fbe3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.200940 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6caf05c79a5f470e8b7796d0e85e36f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.201506 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a978fa28a31b4d589031c77526b62b8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.207227 7528 raft_consensus.cc:493] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.207715 7528 raft_consensus.cc:515] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.160280 7477 ts_tablet_manager.cc:1916] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:57.234230 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2b287c83b77548f19546927b05905746. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.239898 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (DEFAULT_TABLE table=default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.241868 7036 raft_consensus.cc:3060] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.243382 7459 tablet_bootstrap.cc:654] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.245462 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b8fd10368ce242839da23252952d7600" } new_table_name: "default.Table_08bd63c86aca4345bc3ca2746d45456e" modify_external_catalogs: true
I20260317 21:18:57.246367 6973 leader_election.cc:304] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.247617 7037 raft_consensus.cc:3060] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.249179 7528 leader_election.cc:290] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.249646 7528 raft_consensus.cc:493] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.250042 7528 raft_consensus.cc:515] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.255084 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_396735bd5352499580e29038e6317fd4" } new_table_name: "default.Table_d5a32bc495c44a738c19fc1679e42b10" modify_external_catalogs: true
I20260317 21:18:57.252581 7528 leader_election.cc:290] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.282932 7701 raft_consensus.cc:493] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.283066 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:57.283419 7701 raft_consensus.cc:515] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.283756 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c1e672ca556e400ab05763abcd49ad77" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:57.284538 7183 raft_consensus.cc:2393] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:18:57.286388 7701 leader_election.cc:290] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.288326 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 935faf7f9129440a91635bfa96191cd6 with delete_type TABLET_DATA_DELETED (replaced by tablet 57e721ac4b24438b9b2a0e2569027236) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:57.291356 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d (DEFAULT_TABLE table=default.Table_6d4e07837b8945f981be0ae07f5b4ea0 [id=2e0f6bf62c664a6cb3f2bdad9a74e11b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.293462 7301 tablet_service.cc:1511] Processing CreateTablet for tablet fd7cf19121b5496a849118de4a29db5b (DEFAULT_TABLE table=default.Table_713bafc801d244d1a80d9157aebd581b [id=6de336aaef3744f58b4fdfab525c8f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.294009 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a27911e0e545f5811eb075997d0b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.295312 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 63a27911e0e545f5811eb075997d0b0f (DEFAULT_TABLE table=default.Table_ed089cf2f5554ed38d669ac4976ee78c [id=ff741e0270c34ca99f3d1e3e7c45293f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:57.301334 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:57.308900 7459 tablet_bootstrap.cc:492] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:57.309381 7459 ts_tablet_manager.cc:1403] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.157s user 0.011s sys 0.003s
I20260317 21:18:57.317589 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd7cf19121b5496a849118de4a29db5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.322149 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 63a27911e0e545f5811eb075997d0b0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.325554 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59fce28afbeb4d8eadf63f9a97d1183d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.311594 7459 raft_consensus.cc:359] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.326313 7459 raft_consensus.cc:385] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.326591 7459 raft_consensus.cc:740] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.329485 7459 consensus_queue.cc:260] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.334164 7459 ts_tablet_manager.cc:1434] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260317 21:18:57.339061 7528 raft_consensus.cc:2749] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:18:57.344823 7459 tablet_bootstrap.cc:492] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:18:57.349135 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 102850us
W20260317 21:18:57.351562 7120 consensus_peers.cc:597] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be2c469c792047d88fcf2ff9d53b7832. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:57.356077 7672 consensus_queue.cc:1048] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:57.357653 7672 consensus_queue.cc:1048] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:57.360170 7483 ts_tablet_manager.cc:1929] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:57.360759 7483 log.cc:1199] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/c1e672ca556e400ab05763abcd49ad77
I20260317 21:18:57.361490 7483 ts_tablet_manager.cc:1950] T c1e672ca556e400ab05763abcd49ad77 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:57.363564 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 89) took 1748 ms. Trace:
I20260317 21:18:57.367678 7477 ts_tablet_manager.cc:1929] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:57.380344 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:57.381023 7327 raft_consensus.cc:2468] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:57.386358 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b6275af1c9c41ae8e4515deabdbdee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:57.387643 7477 log.cc:1199] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/56495c7cf3964743b96ab885afdaa152
I20260317 21:18:57.388895 6675 master_service.cc:954] ReplaceTablet: received request to replace tablet d2f7c1058f8245428f38fe3245621b75 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:57.389680 7709 raft_consensus.cc:493] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.390107 7477 ts_tablet_manager.cc:1950] T 56495c7cf3964743b96ab885afdaa152 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:57.390249 7709 raft_consensus.cc:515] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.363963 7483 rpcz_store.cc:276] 0317 21:18:55.614973 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.619925 (+ 4952us) service_pool.cc:225] Handling call
0317 21:18:57.191681 (+1571756us) ts_tablet_manager.cc:1151] Deleting tablet c1e672ca556e400ab05763abcd49ad77
0317 21:18:57.191735 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.197619 (+ 5884us) spinlock_profiling.cc:243] Waited 1.14 ms on lock 0x7b5400030cd4. stack: 00007f8cd9137423 00007f8cd9137320 00007f8cd8cac574 0000000000509544 00000000005094d9 00007f8ce01927a1 00007f8ce0231292 00007f8ce065357c 00007f8ce12aa8aa 00007f8ce12db510 00007f8ce12c1add 00007f8ce12c1a89 00007f8ce12c1a19 00007f8ce12c19e1 00007f8ce12c0c0d 00000000005ab084
0317 21:18:57.314325 (+116706us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.359820 (+ 45495us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.363495 (+ 3675us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2616576,"tablet-delete.queue_time_us":1548916}
I20260317 21:18:57.388986 7459 tablet_bootstrap.cc:654] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.392575 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 56495c7cf3964743b96ab885afdaa152 (table default.Table_b1d7ee6663fe432a8940df97de65d99b [id=251402e78beb409192f3bd43ff21a9e6]) successfully deleted
W20260317 21:18:57.383253 6973 leader_election.cc:343] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: c1e672ca556e400ab05763abcd49ad77
I20260317 21:18:57.399973 6973 leader_election.cc:304] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.400848 7705 raft_consensus.cc:3060] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.406224 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 27977b00ae1a4e7ca2eddaa693d11a65 (table default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]) successfully deleted
I20260317 21:18:57.408418 7709 leader_election.cc:290] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:57.411314 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet dc929bae186841d2aba674659e457301 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc929bae186841d2aba674659e457301 already in progress: creating tablet
I20260317 21:18:57.411185 6973 leader_election.cc:304] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.412351 7705 raft_consensus.cc:2749] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:57.412436 7119 leader_election.cc:304] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:57.413748 7709 raft_consensus.cc:2804] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
W20260317 21:18:57.413913 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 935faf7f9129440a91635bfa96191cd6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 935faf7f9129440a91635bfa96191cd6 already in progress: creating tablet
I20260317 21:18:57.414353 7528 raft_consensus.cc:2804] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 21:18:57.414536 7709 raft_consensus.cc:697] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:57.414677 7528 raft_consensus.cc:493] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:57.415019 7528 raft_consensus.cc:3060] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:18:57.416323 7709 consensus_queue.cc:237] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.418491 7459 tablet_bootstrap.cc:492] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:57.418987 7459 ts_tablet_manager.cc:1403] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.010s
I20260317 21:18:57.421587 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 93) took 1225 ms. Trace:
I20260317 21:18:57.421939 7483 rpcz_store.cc:276] 0317 21:18:56.196742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.196888 (+ 146us) service_pool.cc:225] Handling call
0317 21:18:57.393843 (+1196955us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:18:57.393907 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.421519 (+ 27612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1196391}
I20260317 21:18:57.422976 7483 tablet_replica.cc:333] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:57.425006 7528 raft_consensus.cc:515] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.428443 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:57.431129 7483 raft_consensus.cc:2243] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:57.431999 7483 raft_consensus.cc:2272] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:18:57.433565 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet f749358db0f74ef78d9eee23b72bfe3e consensus state has no leader
I20260317 21:18:57.435015 7483 ts_tablet_manager.cc:1916] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:57.441491 7459 raft_consensus.cc:359] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.443416 7459 raft_consensus.cc:385] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.445890 7462 tablet_bootstrap.cc:492] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:57.446308 7459 raft_consensus.cc:740] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.446453 7462 ts_tablet_manager.cc:1403] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.605s user 0.019s sys 0.001s
I20260317 21:18:57.447067 7459 consensus_queue.cc:260] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.457170 7459 ts_tablet_manager.cc:1434] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
W20260317 21:18:57.458333 6662 consensus_peers.cc:597] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 -> Peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Couldn't send request to peer 47e1c983d0224428a9a7b147a0e5e4ef. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111). This is attempt 16: this message will repeat every 5th retry.
I20260317 21:18:57.458654 7459 tablet_bootstrap.cc:492] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:57.466727 7036 raft_consensus.cc:1240] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Rejecting Update request from peer c0ef92fef382401a9564c17e0c223f6a for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260317 21:18:57.478020 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a571e2a5fe2e4b1c92f568eb955c8a38"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:57.479727 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a571e2a5fe2e4b1c92f568eb955c8a38 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:57.481969 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ba0874a8cd746e9a33596ea252b3b14" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:57.482986 7037 raft_consensus.cc:2468] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:57.483223 7528 leader_election.cc:290] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.493577 7580 consensus_queue.cc:1059] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:57.494434 7580 raft_consensus.cc:3055] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Stepping down as leader of term 1
I20260317 21:18:57.495007 7580 raft_consensus.cc:740] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:57.495723 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b8f93eb9b7b047cf9b6604988438ec76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:57.495894 7580 consensus_queue.cc:260] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.497556 7580 raft_consensus.cc:3060] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Advancing to term 2
W20260317 21:18:57.497574 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b8f93eb9b7b047cf9b6604988438ec76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:57.498936 7459 tablet_bootstrap.cc:654] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.502746 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b5a797b22cf34483b0061955da6b2634" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:57.503571 7120 leader_election.cc:304] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:57.504341 7329 raft_consensus.cc:2393] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:18:57.504282 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6d5385171364a078d968bb0a489383a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:57.504482 7672 raft_consensus.cc:2804] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:57.504859 7672 raft_consensus.cc:493] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:57.505170 7672 raft_consensus.cc:3060] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.505427 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "06877bd7155842b9a5ae4a23ea68ada0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:57.527947 7327 raft_consensus.cc:3060] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:18:57.511737 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ba0874a8cd746e9a33596ea252b3b14" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:57.531970 7328 raft_consensus.cc:2468] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:57.537158 7483 ts_tablet_manager.cc:1929] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:57.538230 7327 raft_consensus.cc:2468] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 2.
I20260317 21:18:57.539196 7483 log.cc:1199] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/82f8dd260a134810a50465e4a77848ca
I20260317 21:18:57.540776 7483 ts_tablet_manager.cc:1950] T 82f8dd260a134810a50465e4a77848ca P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:57.550498 6973 leader_election.cc:304] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.551399 7528 raft_consensus.cc:2804] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:18:57.555189 7459 tablet_bootstrap.cc:492] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:57.555680 7459 ts_tablet_manager.cc:1403] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.097s user 0.011s sys 0.000s
I20260317 21:18:57.561636 7528 raft_consensus.cc:697] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:57.563177 7528 consensus_queue.cc:237] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.567860 7459 raft_consensus.cc:359] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.573702 7459 raft_consensus.cc:385] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.574013 7459 raft_consensus.cc:740] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.574725 7459 consensus_queue.cc:260] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.575150 6973 leader_election.cc:304] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.575737 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 94) took 1291 ms. Trace:
I20260317 21:18:57.576125 7483 rpcz_store.cc:276] 0317 21:18:56.283929 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.285878 (+ 1949us) service_pool.cc:225] Handling call
0317 21:18:57.422792 (+1136914us) ts_tablet_manager.cc:1151] Deleting tablet 82f8dd260a134810a50465e4a77848ca
0317 21:18:57.422861 (+ 69us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.450917 (+ 28056us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.536901 (+ 85984us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.575647 (+ 38746us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7381,"spinlock_wait_cycles":3968,"tablet-delete.queue_time_us":1136323}
I20260317 21:18:57.576844 7462 raft_consensus.cc:359] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.577550 7462 raft_consensus.cc:385] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.577847 7462 raft_consensus.cc:740] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.577544 7483 tablet_replica.cc:333] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:57.583405 7483 raft_consensus.cc:2243] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:57.584005 7483 raft_consensus.cc:2272] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.586745 7462 consensus_queue.cc:260] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.589304 7462 ts_tablet_manager.cc:1434] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.143s user 0.007s sys 0.000s
I20260317 21:18:57.592440 7528 raft_consensus.cc:2749] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:57.595332 7459 ts_tablet_manager.cc:1434] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.039s user 0.009s sys 0.000s
I20260317 21:18:57.599658 7672 raft_consensus.cc:515] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:57.600842 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 84738.8us
I20260317 21:18:57.604373 7483 ts_tablet_manager.cc:1916] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:57.610754 7459 tablet_bootstrap.cc:492] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:57.618595 7462 tablet_bootstrap.cc:492] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:57.630368 6674 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet e0875aeb55d34b0bba87a1e97e8e0ebd
I20260317 21:18:57.630581 6673 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 57e721ac4b24438b9b2a0e2569027236
I20260317 21:18:57.631024 6674 master_service.cc:954] ReplaceTablet: received request to replace tablet 589aadc7897b422e90d3f7caed76ab59 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:57.630851 6673 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 160) took 1046 ms. Trace:
I20260317 21:18:57.631381 6673 rpcz_store.cc:276] 0317 21:18:56.584290 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.720249 (+135959us) service_pool.cc:225] Handling call
0317 21:18:56.877500 (+157251us) spinlock_profiling.cc:243] Waited 156 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1386 00007fc05bed7024 00007fc05bed6fb2 00007fc05bed6ee1 00007fc05bed6e5c 00007fc05bed6112 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:56.878827 (+ 1327us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:18:56.879338 (+ 511us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:18:56.879485 (+ 147us) catalog_manager.cc:2824] Locking tablets
0317 21:18:56.975825 (+ 96340us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:18:57.630198 (+654373us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:18:57.630381 (+ 183us) catalog_manager.cc:2872] Committing in-memory state
0317 21:18:57.630504 (+ 123us) catalog_manager.cc:2876] Aborting table tasks
0317 21:18:57.630796 (+ 292us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:18:57.552164 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:18:57.552557 (+ 393us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:18:57.552583 (+ 26us) write_op.cc:435] Acquired schema lock
0317 21:18:57.552595 (+ 12us) tablet.cc:661] Decoding operations
0317 21:18:57.552952 (+ 357us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:18:57.552994 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0317 21:18:57.553017 (+ 23us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:18:57.553282 (+ 265us) tablet.cc:700] Row locks acquired
0317 21:18:57.553297 (+ 15us) write_op.cc:260] PREPARE: finished
0317 21:18:57.553413 (+ 116us) write_op.cc:270] Start()
0317 21:18:57.553501 (+ 88us) write_op.cc:276] Timestamp: P: 1773782337553330 usec, L: 0
0317 21:18:57.553517 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:18:57.554033 (+ 516us) log.cc:844] Serialized 759 byte log entry
0317 21:18:57.627139 (+ 73106us) op_driver.cc:464] REPLICATION: finished
0317 21:18:57.627830 (+ 691us) write_op.cc:301] APPLY: starting
0317 21:18:57.627903 (+ 73us) tablet.cc:1366] starting BulkCheckPresence
0317 21:18:57.628010 (+ 107us) tablet.cc:1369] finished BulkCheckPresence
0317 21:18:57.628112 (+ 102us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:18:57.628400 (+ 288us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:18:57.628545 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:18:57.628590 (+ 45us) write_op.cc:312] APPLY: finished
0317 21:18:57.628825 (+ 235us) log.cc:844] Serialized 29 byte log entry
0317 21:18:57.629475 (+ 650us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:57.629539 (+ 64us) write_op.cc:454] Released schema lock
0317 21:18:57.629570 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":358527872,"child_traces":[["op",{"apply.queue_time_us":496,"num_ops":2,"prepare.queue_time_us":574447,"prepare.run_cpu_time_us":2209,"prepare.run_wall_time_us":28286,"raft.queue_time_us":25979,"raft.run_cpu_time_us":9,"raft.run_wall_time_us":10,"replication_time_us":73510}]]}
W20260317 21:18:57.646574 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 2b287c83b77548f19546927b05905746 because the tablet was not found. No further retry: Not found: Tablet not found: 2b287c83b77548f19546927b05905746
I20260317 21:18:57.650745 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 82f8dd260a134810a50465e4a77848ca (table default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f [id=bcbc5e44c4ce46d2b795f6997fa466ce]) successfully deleted
W20260317 21:18:57.652281 7264 leader_election.cc:343] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:57.652849 7467 tablet_bootstrap.cc:700] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Time spent opening tablet: real 0.136s user 0.000s sys 0.004s
I20260317 21:18:57.653172 7264 leader_election.cc:304] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:57.653510 7467 tablet_bootstrap.cc:654] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.653975 7549 raft_consensus.cc:2749] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:57.663172 7459 tablet_bootstrap.cc:654] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.668346 7038 raft_consensus.cc:3060] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.673507 7478 ts_tablet_manager.cc:1929] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:57.674520 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ba0874a8cd746e9a33596ea252b3b14" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:57.675136 7036 raft_consensus.cc:3060] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.675580 7478 log.cc:1199] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:18:57.676584 7478 ts_tablet_manager.cc:1950] T 53dd9f8a1ccc47f884bfba99781aed16 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:57.677757 7672 leader_election.cc:290] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.687492 7528 raft_consensus.cc:493] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.687992 7528 raft_consensus.cc:515] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.689013 7036 raft_consensus.cc:2468] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:57.690032 7528 leader_election.cc:290] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:57.695034 6670 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 50) took 786 ms (client timeout 1000 ms). Trace:
W20260317 21:18:57.695425 6670 rpcz_store.cc:269] 0317 21:18:56.908508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.912548 (+ 4040us) service_pool.cc:225] Handling call
0317 21:18:57.694962 (+782414us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260317 21:18:57.696156 7120 leader_election.cc:304] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:57.698213 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a7bea1acda68407ea38b257a2659e413"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:57.699705 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7bea1acda68407ea38b257a2659e413 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:57.699544 7580 consensus_queue.cc:1048] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:18:57.701783 7580 raft_consensus.cc:2804] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:57.702279 7580 raft_consensus.cc:697] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:57.703110 7580 consensus_queue.cc:237] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.707168 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99b954ac41134300823458a7a25ef738" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:57.708074 7183 raft_consensus.cc:2468] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:57.711261 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 99b954ac41134300823458a7a25ef738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:57.712037 7477 tablet_replica.cc:333] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:57.712783 7477 raft_consensus.cc:2243] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:57.713240 7477 raft_consensus.cc:2272] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.713452 6971 leader_election.cc:304] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:18:57.714450 6971 raft_consensus.cc:2686] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:18:57.714996 7477 ts_tablet_manager.cc:1916] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:18:57.717100 6973 leader_election.cc:343] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 99b954ac41134300823458a7a25ef738
W20260317 21:18:57.727348 6669 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:45181 (request call id 41) took 900 ms (client timeout 1000 ms). Trace:
W20260317 21:18:57.728150 6669 rpcz_store.cc:269] 0317 21:18:56.827438 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.862428 (+ 34990us) service_pool.cc:225] Handling call
0317 21:18:56.893486 (+ 31058us) spinlock_profiling.cc:243] Waited 28.6 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:57.727276 (+833790us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:18:57.608935 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:18:57.609306 (+ 371us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:18:57.609329 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:18:57.609345 (+ 16us) tablet.cc:661] Decoding operations
0317 21:18:57.609632 (+ 287us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:18:57.609666 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0317 21:18:57.609699 (+ 33us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:18:57.609959 (+ 260us) tablet.cc:700] Row locks acquired
0317 21:18:57.609972 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:18:57.610089 (+ 117us) write_op.cc:270] Start()
0317 21:18:57.610172 (+ 83us) write_op.cc:276] Timestamp: P: 1773782337609996 usec, L: 0
0317 21:18:57.610187 (+ 15us) op_driver.cc:348] REPLICATION: starting
0317 21:18:57.610706 (+ 519us) log.cc:844] Serialized 907 byte log entry
0317 21:18:57.679595 (+ 68889us) op_driver.cc:464] REPLICATION: finished
0317 21:18:57.679832 (+ 237us) write_op.cc:301] APPLY: starting
0317 21:18:57.679888 (+ 56us) tablet.cc:1366] starting BulkCheckPresence
0317 21:18:57.680000 (+ 112us) tablet.cc:1369] finished BulkCheckPresence
0317 21:18:57.680085 (+ 85us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:18:57.680337 (+ 252us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:18:57.680474 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:18:57.680523 (+ 49us) write_op.cc:312] APPLY: finished
0317 21:18:57.680752 (+ 229us) log.cc:844] Serialized 29 byte log entry
0317 21:18:57.680904 (+ 152us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:57.680980 (+ 76us) write_op.cc:454] Released schema lock
0317 21:18:57.681013 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0317 21:18:57.681236 (+ 223us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:57.681263 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":6,"spinlock_wait_cycles":65730432,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":1372,"apply.run_wall_time_us":1371,"num_ops":2,"prepare.queue_time_us":546298,"prepare.run_cpu_time_us":2157,"prepare.run_wall_time_us":28334,"raft.queue_time_us":14672,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":19,"replication_time_us":69319}]]}
I20260317 21:18:57.740991 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_94ac54eae69d40e6974b829a10d006ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:57.664474 7462 tablet_bootstrap.cc:654] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:57.743597 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94ac54eae69d40e6974b829a10d006ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:57.741025 6671 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet e858a01bc4594cc58853ea3239a8dbad (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]). Expected version 1 got 0
I20260317 21:18:57.746346 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 99b954ac41134300823458a7a25ef738 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:18:57.610090 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 99544.2us
W20260317 21:18:57.765890 7119 consensus_peers.cc:597] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: be2c469c792047d88fcf2ff9d53b7832. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:57.765865 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8ba0874a8cd746e9a33596ea252b3b14" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:57.768594 7329 raft_consensus.cc:1275] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Refusing update from remote peer c0ef92fef382401a9564c17e0c223f6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:18:57.768625 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 80) took 2886 ms. Trace:
I20260317 21:18:57.778151 7478 rpcz_store.cc:276] 0317 21:18:54.882659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:54.893719 (+ 11060us) service_pool.cc:225] Handling call
0317 21:18:56.807943 (+1914224us) ts_tablet_manager.cc:1151] Deleting tablet 53dd9f8a1ccc47f884bfba99781aed16
0317 21:18:56.807993 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.586693 (+778700us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.673267 (+ 86574us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.679680 (+ 6413us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1913600}
I20260317 21:18:57.779585 7478 tablet_replica.cc:333] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:57.780375 7478 raft_consensus.cc:2243] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:57.781652 7478 raft_consensus.cc:2272] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.783515 7478 ts_tablet_manager.cc:1916] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:57.786147 7459 tablet_bootstrap.cc:492] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:57.786643 7459 ts_tablet_manager.cc:1403] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.176s user 0.012s sys 0.005s
I20260317 21:18:57.797654 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 53dd9f8a1ccc47f884bfba99781aed16 (table default.Table_afb9c6074ee54f86a4d029cf8cdbb825 [id=dd080d09b8324bcabbe022d42352fadb]) successfully deleted
I20260317 21:18:57.804140 7459 raft_consensus.cc:359] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:57.792865 7119 leader_election.cc:343] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:57.816582 7717 consensus_queue.cc:1048] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:18:57.817282 7459 raft_consensus.cc:385] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.817829 7459 raft_consensus.cc:740] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.818598 7459 consensus_queue.cc:260] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.794148 7580 consensus_queue.cc:1048] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:57.802256 7467 tablet_bootstrap.cc:492] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:57.820497 7467 ts_tablet_manager.cc:1403] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.770s user 0.008s sys 0.007s
I20260317 21:18:57.823139 7467 raft_consensus.cc:359] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.823791 7467 raft_consensus.cc:385] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.824050 7467 raft_consensus.cc:740] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.826292 7459 ts_tablet_manager.cc:1434] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260317 21:18:57.828972 7467 consensus_queue.cc:260] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.831234 7467 ts_tablet_manager.cc:1434] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.010s user 0.005s sys 0.000s
I20260317 21:18:57.832401 7467 tablet_bootstrap.cc:492] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:57.832412 7477 ts_tablet_manager.cc:1929] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:57.841028 7477 log.cc:1199] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/99b954ac41134300823458a7a25ef738
I20260317 21:18:57.838013 7459 tablet_bootstrap.cc:492] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:57.846151 7477 ts_tablet_manager.cc:1950] T 99b954ac41134300823458a7a25ef738 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:57.851191 7467 tablet_bootstrap.cc:654] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.862813 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 99b954ac41134300823458a7a25ef738 (table default.Table_93723f1459114d9ba82a21cee9158b3d [id=d079fd8ed7aa48ad9420c7cf7419c36e]) successfully deleted
I20260317 21:18:57.877961 7528 raft_consensus.cc:493] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:57.878470 7528 raft_consensus.cc:515] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.878700 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_fcec6e585d6245ff828c8c0c98450d63" } modify_external_catalogs: true
I20260317 21:18:57.879290 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_fcec6e585d6245ff828c8c0c98450d63" } modify_external_catalogs: true
I20260317 21:18:57.880550 7528 leader_election.cc:290] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:57.888408 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:57.889617 7327 raft_consensus.cc:2468] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:57.889516 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:57.891525 6971 leader_election.cc:343] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:57.892184 7459 tablet_bootstrap.cc:654] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:57.892059 6973 leader_election.cc:304] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.893082 7528 raft_consensus.cc:2804] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:57.893452 7528 raft_consensus.cc:493] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:57.893858 7528 raft_consensus.cc:3060] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.901517 7478 ts_tablet_manager.cc:1929] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:57.902065 7478 log.cc:1199] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/be2c469c792047d88fcf2ff9d53b7832
I20260317 21:18:57.902907 7478 ts_tablet_manager.cc:1950] T be2c469c792047d88fcf2ff9d53b7832 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:57.903434 7528 raft_consensus.cc:515] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.909591 7740 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:18:57.918478 7462 tablet_bootstrap.cc:492] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:57.918953 7462 ts_tablet_manager.cc:1403] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.301s user 0.004s sys 0.008s
I20260317 21:18:57.926239 7483 ts_tablet_manager.cc:1929] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:57.926983 7459 tablet_bootstrap.cc:492] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:57.927433 7459 ts_tablet_manager.cc:1403] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.003s
I20260317 21:18:57.929986 7459 raft_consensus.cc:359] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.930627 7459 raft_consensus.cc:385] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.930867 7459 raft_consensus.cc:740] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.906153 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 83) took 2860 ms. Trace:
I20260317 21:18:57.933219 7478 rpcz_store.cc:276] 0317 21:18:55.045584 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.056034 (+ 10450us) service_pool.cc:225] Handling call
0317 21:18:57.779421 (+2723387us) ts_tablet_manager.cc:1151] Deleting tablet be2c469c792047d88fcf2ff9d53b7832
0317 21:18:57.779473 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.859288 (+ 79815us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.901257 (+ 41969us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.906072 (+ 4815us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2722307}
I20260317 21:18:57.936228 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:18:57.939874 6971 leader_election.cc:343] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:57.940371 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet be2c469c792047d88fcf2ff9d53b7832 (table default.Table_ffede0d29ebf494f8eec15158eeea6f6 [id=e9385f5ddf6e430a83595a74932a704e]) successfully deleted
I20260317 21:18:57.941481 7478 tablet_replica.cc:333] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:57.942276 7459 consensus_queue.cc:260] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.947227 7478 raft_consensus.cc:2243] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:57.948025 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:57.948734 7327 raft_consensus.cc:3060] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:57.949847 7478 raft_consensus.cc:2272] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:57.951762 7478 ts_tablet_manager.cc:1916] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:18:57.926610 7734 log.cc:927] Time spent T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: Append to log took a long time: real 0.053s user 0.001s sys 0.000s
I20260317 21:18:57.968977 7459 ts_tablet_manager.cc:1434] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.041s user 0.004s sys 0.005s
I20260317 21:18:57.948766 7742 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:18:57.969033 7740 tablet.cc:1723] T e1094068903840a8ab5d07c7fb0635bf P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:57.970500 7741 tablet.cc:1723] T e1094068903840a8ab5d07c7fb0635bf P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:57.957818 7327 raft_consensus.cc:2468] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:57.975386 6973 leader_election.cc:304] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:57.976154 7705 raft_consensus.cc:2804] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:57.927071 7483 log.cc:1199] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:18:57.983012 7483 ts_tablet_manager.cc:1950] T 8ba0874a8cd746e9a33596ea252b3b14 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:57.983506 7467 tablet_bootstrap.cc:492] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:57.984009 7467 ts_tablet_manager.cc:1403] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.152s user 0.022s sys 0.020s
I20260317 21:18:57.985996 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet d2f7c1058f8245428f38fe3245621b75
I20260317 21:18:57.986073 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 95) took 1698 ms. Trace:
I20260317 21:18:57.986370 7483 rpcz_store.cc:276] 0317 21:18:56.288153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.288264 (+ 111us) service_pool.cc:225] Handling call
0317 21:18:57.577369 (+1289105us) ts_tablet_manager.cc:1151] Deleting tablet 8ba0874a8cd746e9a33596ea252b3b14
0317 21:18:57.577416 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:57.695998 (+118582us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.925822 (+229824us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:57.986007 (+ 60185us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288545}
I20260317 21:18:57.986649 7467 raft_consensus.cc:359] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.987267 7467 raft_consensus.cc:385] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:57.987551 7483 tablet_replica.cc:333] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:57.987520 6675 catalog_manager.cc:6741] ReplaceTablet: tablet d2f7c1058f8245428f38fe3245621b75 deleted and replaced by tablet e22772281bd74754bb969e87a70d8077
I20260317 21:18:57.987506 7467 raft_consensus.cc:740] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:57.988202 7483 raft_consensus.cc:2243] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:57.994071 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet d2f7c1058f8245428f38fe3245621b75 with delete_type TABLET_DATA_DELETED (replaced by tablet e22772281bd74754bb969e87a70d8077) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:57.994772 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet d2f7c1058f8245428f38fe3245621b75 with delete_type TABLET_DATA_DELETED (replaced by tablet e22772281bd74754bb969e87a70d8077) from {username='slave'} at 127.0.0.1:47442
W20260317 21:18:57.996605 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet d2f7c1058f8245428f38fe3245621b75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2f7c1058f8245428f38fe3245621b75 already in progress: creating tablet
I20260317 21:18:57.997624 7744 tablet.cc:1723] T e1094068903840a8ab5d07c7fb0635bf P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:18:58.001140 7705 raft_consensus.cc:697] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:57.988087 7467 consensus_queue.cc:260] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.002660 7705 consensus_queue.cc:237] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.008620 7459 tablet_bootstrap.cc:492] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:58.010151 7709 raft_consensus.cc:493] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.010624 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_45189d08af7842588601921563fa3b41" } new_table_name: "default.Table_1c93ac734b4d4ff68afbd9c72d470944" modify_external_catalogs: true
I20260317 21:18:58.010739 7709 raft_consensus.cc:515] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:57.988631 7483 raft_consensus.cc:2272] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.013161 7467 ts_tablet_manager.cc:1434] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.029s user 0.004s sys 0.004s
I20260317 21:18:58.014168 7467 tablet_bootstrap.cc:492] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:58.014321 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:58.016614 7120 leader_election.cc:343] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:58.013190 7709 leader_election.cc:290] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.018687 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d5a32bc495c44a738c19fc1679e42b10" } modify_external_catalogs: true
I20260317 21:18:58.020037 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_efd3fbe665cd4ca4ab6be9a618c17a6b" } new_table_name: "default.Table_3ba355ed208a4665a3533866f1bc7d38" modify_external_catalogs: true
I20260317 21:18:58.021487 7709 raft_consensus.cc:493] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.022034 7709 raft_consensus.cc:515] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.024369 7709 leader_election.cc:290] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.026144 7483 ts_tablet_manager.cc:1916] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.026921 7467 tablet_bootstrap.cc:654] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.029968 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d5a32bc495c44a738c19fc1679e42b10" } modify_external_catalogs: true
I20260317 21:18:58.033912 7747 tablet.cc:1723] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:18:58.035445 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60f3d32e1f7649fa92861f51134cf000" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:58.036986 7120 leader_election.cc:343] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:58.042191 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 8ba0874a8cd746e9a33596ea252b3b14 (table default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]) successfully deleted
I20260317 21:18:58.047302 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet d2f7c1058f8245428f38fe3245621b75 with delete_type TABLET_DATA_DELETED (replaced by tablet e22772281bd74754bb969e87a70d8077) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:58.050628 7459 tablet_bootstrap.cc:654] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.071206 7745 raft_consensus.cc:493] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.071862 7745 raft_consensus.cc:515] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:58.074798 6662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20260317 21:18:58.077353 7745 leader_election.cc:290] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.078281 7467 tablet_bootstrap.cc:492] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
W20260317 21:18:58.078419 6971 leader_election.cc:343] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:18:58.078926 7467 ts_tablet_manager.cc:1403] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.065s user 0.015s sys 0.004s
I20260317 21:18:58.085163 7462 raft_consensus.cc:359] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.085955 7462 raft_consensus.cc:385] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.086261 7462 raft_consensus.cc:740] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.086934 7462 consensus_queue.cc:260] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.099848 7467 raft_consensus.cc:359] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.100525 7467 raft_consensus.cc:385] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.100802 7467 raft_consensus.cc:740] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.101366 7467 consensus_queue.cc:260] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.103325 7467 ts_tablet_manager.cc:1434] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.024s user 0.003s sys 0.000s
I20260317 21:18:58.070062 7528 leader_election.cc:290] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.121336 7459 tablet_bootstrap.cc:492] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:58.122004 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:58.122843 7327 raft_consensus.cc:2468] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:58.122917 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60f3d32e1f7649fa92861f51134cf000" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:58.125587 7119 leader_election.cc:304] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:58.127063 7467 tablet_bootstrap.cc:492] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:58.128968 7717 raft_consensus.cc:2804] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:58.129410 7462 ts_tablet_manager.cc:1434] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.210s user 0.006s sys 0.001s
I20260317 21:18:58.129428 7717 raft_consensus.cc:493] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 21:18:58.129671 6973 leader_election.cc:343] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:18:58.129905 7717 raft_consensus.cc:3060] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.130102 6973 leader_election.cc:304] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:18:58.131042 7528 raft_consensus.cc:2749] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:58.132097 7459 ts_tablet_manager.cc:1403] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.124s user 0.006s sys 0.010s
I20260317 21:18:58.135092 7459 raft_consensus.cc:359] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:58.138932 7119 leader_election.cc:343] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:18:58.139387 7119 leader_election.cc:304] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:58.142176 7709 raft_consensus.cc:2749] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:18:58.145591 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3d40549347f8463797d6791136883371 has no consensus state
W20260317 21:18:58.147114 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 275e9cd7f14040d98eb85a6bb1010c2c has no consensus state
I20260317 21:18:58.148243 7459 raft_consensus.cc:385] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.148581 7459 raft_consensus.cc:740] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.149345 7459 consensus_queue.cc:260] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.156834 7478 ts_tablet_manager.cc:1929] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:58.161629 7462 tablet_bootstrap.cc:492] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:58.163797 7459 ts_tablet_manager.cc:1434] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.031s user 0.002s sys 0.003s
I20260317 21:18:58.165262 7478 log.cc:1199] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/27977b00ae1a4e7ca2eddaa693d11a65
I20260317 21:18:58.166198 7478 ts_tablet_manager.cc:1950] T 27977b00ae1a4e7ca2eddaa693d11a65 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.163583 7717 raft_consensus.cc:515] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.172000 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:18:58.173025 7459 tablet_bootstrap.cc:492] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:18:58.173796 7120 leader_election.cc:343] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:58.175786 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 019d90f6d70e46b087ea3c5ffd7cbd28
I20260317 21:18:58.176313 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:58.176926 7329 raft_consensus.cc:3060] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.179378 7459 tablet_bootstrap.cc:654] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.179325 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:58 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:58.180272 7477 tablet_replica.cc:333] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:58.180349 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:58 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:58.180982 7477 raft_consensus.cc:2243] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.181468 7477 raft_consensus.cc:2272] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.184051 7477 ts_tablet_manager.cc:1916] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.185683 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:58 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:58.188024 7329 raft_consensus.cc:2468] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:58.188562 7717 leader_election.cc:290] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.190248 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 87) took 2775 ms. Trace:
I20260317 21:18:58.197477 7478 rpcz_store.cc:276] 0317 21:18:55.415182 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.415382 (+ 200us) service_pool.cc:225] Handling call
0317 21:18:57.941276 (+2525894us) ts_tablet_manager.cc:1151] Deleting tablet 27977b00ae1a4e7ca2eddaa693d11a65
0317 21:18:57.941335 (+ 59us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.087859 (+146524us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.156558 (+ 68699us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.190181 (+ 33623us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2525251}
I20260317 21:18:58.199272 7483 ts_tablet_manager.cc:1929] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.199673 7483 log.cc:1199] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/60f3d32e1f7649fa92861f51134cf000
I20260317 21:18:58.199393 7119 leader_election.cc:304] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:58.200368 7483 ts_tablet_manager.cc:1950] T 60f3d32e1f7649fa92861f51134cf000 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:58.200992 7717 raft_consensus.cc:2804] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:58.202493 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 97) took 1882 ms. Trace:
I20260317 21:18:58.205690 7483 rpcz_store.cc:276] 0317 21:18:56.319692 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.319790 (+ 98us) service_pool.cc:225] Handling call
0317 21:18:57.987399 (+1667609us) ts_tablet_manager.cc:1151] Deleting tablet 60f3d32e1f7649fa92861f51134cf000
0317 21:18:57.987440 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.105671 (+118231us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.199033 (+ 93362us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.202423 (+ 3390us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1667114}
I20260317 21:18:58.207652 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 96) took 1889 ms. Trace:
I20260317 21:18:58.208027 7483 rpcz_store.cc:276] 0317 21:18:56.318439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.318951 (+ 512us) service_pool.cc:225] Handling call
0317 21:18:58.207281 (+1888330us) ts_tablet_manager.cc:1151] Deleting tablet 53dd9f8a1ccc47f884bfba99781aed16
0317 21:18:58.207331 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.207575 (+ 244us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886842}
I20260317 21:18:58.209858 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_532570073e77485ca4f7d839c89cc71f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.212026 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 98) took 1837 ms. Trace:
W20260317 21:18:58.211865 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_532570073e77485ca4f7d839c89cc71f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.212379 7483 rpcz_store.cc:276] 0317 21:18:56.374445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.374588 (+ 143us) service_pool.cc:225] Handling call
0317 21:18:58.209096 (+1834508us) ts_tablet_manager.cc:1151] Deleting tablet 163cc8c04ca94a269c7ae2efaa162525
0317 21:18:58.211744 (+ 2648us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.211964 (+ 220us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832506}
I20260317 21:18:58.212945 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 60f3d32e1f7649fa92861f51134cf000 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]) successfully deleted
I20260317 21:18:58.213479 7483 tablet_replica.cc:333] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
W20260317 21:18:58.213786 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 53dd9f8a1ccc47f884bfba99781aed16 because the tablet was not found. No further retry: Not found: Tablet not found: 53dd9f8a1ccc47f884bfba99781aed16
W20260317 21:18:58.214370 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 163cc8c04ca94a269c7ae2efaa162525 because the tablet was not found. No further retry: Not found: Tablet not found: 163cc8c04ca94a269c7ae2efaa162525
I20260317 21:18:58.208045 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_29eda40c73594995aebde51a209340f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:58.218113 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29eda40c73594995aebde51a209340f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.220952 7717 raft_consensus.cc:697] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:58.221228 7483 raft_consensus.cc:2243] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.221737 7483 raft_consensus.cc:2272] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.221984 7717 consensus_queue.cc:237] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.223609 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 27977b00ae1a4e7ca2eddaa693d11a65 (table default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]) successfully deleted
I20260317 21:18:58.223764 7483 ts_tablet_manager.cc:1916] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.228973 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 86) took 2809 ms. Trace:
I20260317 21:18:58.229461 7478 rpcz_store.cc:276] 0317 21:18:55.414657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.423117 (+ 8460us) service_pool.cc:225] Handling call
0317 21:18:58.223679 (+2800562us) ts_tablet_manager.cc:1151] Deleting tablet b6fb7cb082cf47ce8cf2a98146633af7
0317 21:18:58.223727 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.224006 (+ 279us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2799894}
W20260317 21:18:58.230815 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet b6fb7cb082cf47ce8cf2a98146633af7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b6fb7cb082cf47ce8cf2a98146633af7 already in progress: creating tablet
I20260317 21:18:58.232309 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 88) took 2727 ms. Trace:
I20260317 21:18:58.232775 7478 rpcz_store.cc:276] 0317 21:18:55.505123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:55.505274 (+ 151us) service_pool.cc:225] Handling call
0317 21:18:58.230874 (+2725600us) ts_tablet_manager.cc:1151] Deleting tablet dc929bae186841d2aba674659e457301
0317 21:18:58.230916 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.232253 (+ 1337us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2724485}
I20260317 21:18:58.232909 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_8ca6f39801634869ae519a57be7cfd69"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:58.234655 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ca6f39801634869ae519a57be7cfd69 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.236997 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 89) took 2042 ms. Trace:
I20260317 21:18:58.237430 7478 rpcz_store.cc:276] 0317 21:18:56.194742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.194901 (+ 159us) service_pool.cc:225] Handling call
0317 21:18:58.236598 (+2041697us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:18:58.236637 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.236941 (+ 304us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2041005}
W20260317 21:18:58.237830 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet dc929bae186841d2aba674659e457301 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet dc929bae186841d2aba674659e457301 already in progress: creating tablet
W20260317 21:18:58.241375 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 2b287c83b77548f19546927b05905746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b287c83b77548f19546927b05905746 already in progress: creating tablet
I20260317 21:18:58.242830 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 90) took 1959 ms. Trace:
I20260317 21:18:58.243206 7478 rpcz_store.cc:276] 0317 21:18:56.283480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.283630 (+ 150us) service_pool.cc:225] Handling call
0317 21:18:58.242485 (+1958855us) ts_tablet_manager.cc:1151] Deleting tablet 82f8dd260a134810a50465e4a77848ca
0317 21:18:58.242524 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.242774 (+ 250us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1958202}
I20260317 21:18:58.243832 6670 master_service.cc:954] ReplaceTablet: received request to replace tablet 06877bd7155842b9a5ae4a23ea68ada0 from {username='slave'} at 127.0.0.1:39038
W20260317 21:18:58.244460 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 82f8dd260a134810a50465e4a77848ca because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 82f8dd260a134810a50465e4a77848ca already in progress: creating tablet
I20260317 21:18:58.245390 7478 tablet_replica.cc:333] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.247296 7462 tablet_bootstrap.cc:654] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.248126 7478 raft_consensus.cc:2243] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:58.249110 7478 pending_rounds.cc:70] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:18:58.252383 7483 ts_tablet_manager.cc:1929] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.257216 7478 pending_rounds.cc:77] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412455241379840 op_type: NO_OP noop_request { }
I20260317 21:18:58.262185 7478 raft_consensus.cc:2889] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:18:58.262770 7478 raft_consensus.cc:2272] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.265138 7478 ts_tablet_manager.cc:1916] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.276158 7459 tablet_bootstrap.cc:492] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:58.276666 7459 ts_tablet_manager.cc:1403] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.104s user 0.006s sys 0.009s
I20260317 21:18:58.279034 7459 raft_consensus.cc:359] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.279687 7459 raft_consensus.cc:385] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.279958 7459 raft_consensus.cc:740] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.287014 7467 tablet_bootstrap.cc:654] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.268378 7736 raft_consensus.cc:493] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.296473 7736 raft_consensus.cc:515] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.298851 7736 leader_election.cc:290] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:58.299371 7483 log.cc:1199] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/d59deea79cfb4a83831bb0227797c690
I20260317 21:18:58.299685 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2f7c1058f8245428f38fe3245621b75" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:58.300137 7483 ts_tablet_manager.cc:1950] T d59deea79cfb4a83831bb0227797c690 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
W20260317 21:18:58.301121 7264 leader_election.cc:343] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:58.301254 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2f7c1058f8245428f38fe3245621b75" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:58.297109 7478 ts_tablet_manager.cc:1929] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:58.301859 7478 log.cc:1199] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:18:58.302037 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 99) took 1918 ms. Trace:
W20260317 21:18:58.302590 7261 leader_election.cc:343] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:58.303076 7261 leader_election.cc:304] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:58.302325 7483 rpcz_store.cc:276] 0317 21:18:56.383195 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.383316 (+ 121us) service_pool.cc:225] Handling call
0317 21:18:58.213317 (+1830001us) ts_tablet_manager.cc:1151] Deleting tablet d59deea79cfb4a83831bb0227797c690
0317 21:18:58.213362 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.244139 (+ 30777us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.252149 (+ 8010us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.301982 (+ 49833us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6849,"tablet-delete.queue_time_us":1829420}
I20260317 21:18:58.304201 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 100) took 1882 ms. Trace:
I20260317 21:18:58.305439 7478 ts_tablet_manager.cc:1950] T 8ba0874a8cd746e9a33596ea252b3b14 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.306875 7477 ts_tablet_manager.cc:1929] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260317 21:18:58.307255 7477 log.cc:1199] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/019d90f6d70e46b087ea3c5ffd7cbd28
I20260317 21:18:58.307933 7477 ts_tablet_manager.cc:1950] T 019d90f6d70e46b087ea3c5ffd7cbd28 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:18:58.304184 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet d59deea79cfb4a83831bb0227797c690 (table default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]) successfully deleted
I20260317 21:18:58.304772 7736 raft_consensus.cc:2749] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:58.312047 7483 rpcz_store.cc:276] 0317 21:18:56.421711 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.421889 (+ 178us) service_pool.cc:225] Handling call
0317 21:18:58.303838 (+1881949us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:18:58.303888 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.304148 (+ 260us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1880891}
I20260317 21:18:58.312767 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_ef1aaeadf5354f7e99d9db422dfc6e63"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.313345 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 105) took 1544 ms. Trace:
W20260317 21:18:58.313442 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 2b287c83b77548f19546927b05905746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b287c83b77548f19546927b05905746 already in progress: creating tablet
I20260317 21:18:58.313628 7483 rpcz_store.cc:276] 0317 21:18:56.769213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.769411 (+ 198us) service_pool.cc:225] Handling call
0317 21:18:58.313088 (+1543677us) ts_tablet_manager.cc:1151] Deleting tablet b6fb7cb082cf47ce8cf2a98146633af7
0317 21:18:58.313129 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.313299 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1543097}
W20260317 21:18:58.314253 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ef1aaeadf5354f7e99d9db422dfc6e63 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:58.314827 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet b6fb7cb082cf47ce8cf2a98146633af7 because the tablet was not found. No further retry: Not found: Tablet not found: b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:18:58.315763 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 91) took 2010 ms. Trace:
I20260317 21:18:58.316078 7478 rpcz_store.cc:276] 0317 21:18:56.305580 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.305737 (+ 157us) service_pool.cc:225] Handling call
0317 21:18:58.245247 (+1939510us) ts_tablet_manager.cc:1151] Deleting tablet 8ba0874a8cd746e9a33596ea252b3b14
0317 21:18:58.245286 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.282177 (+ 36891us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.296861 (+ 14684us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.315697 (+ 18836us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1938963}
I20260317 21:18:58.317221 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 8ba0874a8cd746e9a33596ea252b3b14 (table default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]) successfully deleted
I20260317 21:18:58.323119 7483 tablet_replica.cc:333] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:58.320305 7459 consensus_queue.cc:260] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.323926 7483 raft_consensus.cc:2243] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.324595 7483 raft_consensus.cc:2272] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.326373 7483 ts_tablet_manager.cc:1916] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.327780 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 92) took 2003 ms. Trace:
I20260317 21:18:58.338757 7717 consensus_queue.cc:1048] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:58.347373 7717 raft_consensus.cc:493] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.347849 7717 raft_consensus.cc:515] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.349578 7483 ts_tablet_manager.cc:1929] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.350008 7483 log.cc:1199] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/c6d5385171364a078d968bb0a489383a
I20260317 21:18:58.350099 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_d5a32bc495c44a738c19fc1679e42b10 [id=2ed77cec958e4b008989a60fb67a2adc]) successfully deleted
I20260317 21:18:58.350306 7717 leader_election.cc:290] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:18:58.352133 7119 leader_election.cc:343] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: d59deea79cfb4a83831bb0227797c690
I20260317 21:18:58.352353 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d59deea79cfb4a83831bb0227797c690" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:58.352977 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0c38032670874e6f85493a54b207749e" } new_table_name: "default.Table_c037d6417fdf415cb20c3bb9c342c07d" modify_external_catalogs: true
I20260317 21:18:58.354672 7120 leader_election.cc:304] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:58.357908 7580 raft_consensus.cc:2749] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:58.358482 7483 ts_tablet_manager.cc:1950] T c6d5385171364a078d968bb0a489383a P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:58.359908 7329 raft_consensus.cc:1275] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Refusing update from remote peer d3cb3b49e3e94537a6e434ea4e037c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 21:18:58.356504 7467 tablet_bootstrap.cc:492] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:58.360352 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 110) took 1289 ms. Trace:
I20260317 21:18:58.360575 7467 ts_tablet_manager.cc:1403] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.234s user 0.010s sys 0.004s
I20260317 21:18:58.361830 7705 consensus_queue.cc:1048] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:58.328118 7478 rpcz_store.cc:276] 0317 21:18:56.324117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.324270 (+ 153us) service_pool.cc:225] Handling call
0317 21:18:58.327412 (+2003142us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:18:58.327454 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.327719 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002122}
I20260317 21:18:58.363787 7478 tablet_replica.cc:333] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.364254 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4d068d09b4944465bfdf018355207dc3" } new_table_name: "default.Table_4088537e0de64ecc8cba0ebff9c50bbc" modify_external_catalogs: true
I20260317 21:18:58.364507 7478 raft_consensus.cc:2243] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.364975 7478 raft_consensus.cc:2272] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.369848 7483 rpcz_store.cc:276] 0317 21:18:57.070673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.071153 (+ 480us) service_pool.cc:225] Handling call
0317 21:18:58.322948 (+1251795us) ts_tablet_manager.cc:1151] Deleting tablet c6d5385171364a078d968bb0a489383a
0317 21:18:58.322992 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.340036 (+ 17044us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.349342 (+ 9306us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.360291 (+ 10949us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1248541}
I20260317 21:18:58.378211 7478 ts_tablet_manager.cc:1916] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.379957 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet c6d5385171364a078d968bb0a489383a (table default.Table_4493a733fd0a496885b7745fcfc6cf2e [id=e8d1164df252476a82baf64ec8c6c9cf]) successfully deleted
I20260317 21:18:58.363397 7467 raft_consensus.cc:359] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.381667 7467 raft_consensus.cc:385] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.381705 7483 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 114) took 1097 ms. Trace:
I20260317 21:18:58.381937 7467 raft_consensus.cc:740] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.382002 7483 rpcz_store.cc:276] 0317 21:18:57.284788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.287969 (+ 3181us) service_pool.cc:225] Handling call
0317 21:18:58.379810 (+1091841us) ts_tablet_manager.cc:1151] Deleting tablet 935faf7f9129440a91635bfa96191cd6
0317 21:18:58.381402 (+ 1592us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.381651 (+ 249us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090067}
I20260317 21:18:58.383076 7483 tablet_replica.cc:333] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:58.383013 7467 consensus_queue.cc:260] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.383713 7483 raft_consensus.cc:2243] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.384142 7483 raft_consensus.cc:2272] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.385535 7467 ts_tablet_manager.cc:1434] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260317 21:18:58.386978 7459 ts_tablet_manager.cc:1434] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.110s user 0.006s sys 0.000s
I20260317 21:18:58.387270 7467 tablet_bootstrap.cc:492] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:18:58.389559 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eb332f981fc84a178fadae2f4908bdde consensus state has no leader
I20260317 21:18:58.393468 7183 raft_consensus.cc:1275] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Refusing update from remote peer d3cb3b49e3e94537a6e434ea4e037c3f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260317 21:18:58.393841 7183 pending_rounds.cc:85] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Aborting all ops after (but not including) 0
I20260317 21:18:58.394093 7183 pending_rounds.cc:107] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260317 21:18:58.394340 7183 raft_consensus.cc:2889] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:18:58.398625 7462 tablet_bootstrap.cc:492] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:58.399461 7462 ts_tablet_manager.cc:1403] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.238s user 0.011s sys 0.001s
I20260317 21:18:58.400620 7745 consensus_queue.cc:1048] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:18:58.364378 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 2b287c83b77548f19546927b05905746 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2b287c83b77548f19546927b05905746 already in progress: creating tablet
I20260317 21:18:58.407629 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_20cf24578433433a96ad72f0593ca863" } modify_external_catalogs: true
I20260317 21:18:58.408227 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_20cf24578433433a96ad72f0593ca863" } modify_external_catalogs: true
W20260317 21:18:58.415337 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a978fa28a31b4d589031c77526b62b8e consensus state has no leader
I20260317 21:18:58.439903 7459 tablet_bootstrap.cc:492] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:18:58.442075 6662 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet eb332f981fc84a178fadae2f4908bdde consensus state has no leader
I20260317 21:18:58.452461 7478 ts_tablet_manager.cc:1929] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.452929 7478 log.cc:1199] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/d59deea79cfb4a83831bb0227797c690
I20260317 21:18:58.454536 7478 ts_tablet_manager.cc:1950] T d59deea79cfb4a83831bb0227797c690 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.456530 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 93) took 2118 ms. Trace:
I20260317 21:18:58.456884 7478 rpcz_store.cc:276] 0317 21:18:56.337983 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.343008 (+ 5025us) service_pool.cc:225] Handling call
0317 21:18:58.363639 (+2020631us) ts_tablet_manager.cc:1151] Deleting tablet d59deea79cfb4a83831bb0227797c690
0317 21:18:58.363679 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.432432 (+ 68753us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.451187 (+ 18755us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.456471 (+ 5284us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2020047}
I20260317 21:18:58.457985 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet d59deea79cfb4a83831bb0227797c690 (table default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]) successfully deleted
I20260317 21:18:58.458863 7478 tablet_replica.cc:333] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.459709 7478 raft_consensus.cc:2243] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:58.460543 7478 raft_consensus.cc:2272] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.464179 7462 raft_consensus.cc:359] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.465242 7462 raft_consensus.cc:385] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.465546 7462 raft_consensus.cc:740] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.468354 7483 ts_tablet_manager.cc:1916] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.471031 7462 consensus_queue.cc:260] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.473176 7459 tablet_bootstrap.cc:654] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.479764 7478 ts_tablet_manager.cc:1916] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.453917 7467 tablet_bootstrap.cc:654] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.500897 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_8f61c0b1dfa846baa3c30adcbba8b683"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.501309 7462 ts_tablet_manager.cc:1434] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.102s user 0.001s sys 0.007s
I20260317 21:18:58.501163 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_81205ffa8e804950ae9aff3e9806bb0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:58.502830 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_81205ffa8e804950ae9aff3e9806bb0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:58.503249 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8f61c0b1dfa846baa3c30adcbba8b683 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.505788 7478 ts_tablet_manager.cc:1929] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:18:58.506218 7478 log.cc:1199] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/c1e672ca556e400ab05763abcd49ad77
I20260317 21:18:58.506953 7478 ts_tablet_manager.cc:1950] T c1e672ca556e400ab05763abcd49ad77 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.508947 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 99) took 1751 ms. Trace:
I20260317 21:18:58.509258 7478 rpcz_store.cc:276] 0317 21:18:56.758001 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.758127 (+ 126us) service_pool.cc:225] Handling call
0317 21:18:58.458677 (+1700550us) ts_tablet_manager.cc:1151] Deleting tablet c1e672ca556e400ab05763abcd49ad77
0317 21:18:58.458725 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.488955 (+ 30230us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.505545 (+ 16590us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.508892 (+ 3347us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1699939}
I20260317 21:18:58.510316 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet c1e672ca556e400ab05763abcd49ad77 (table default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]) successfully deleted
I20260317 21:18:58.510578 7478 tablet_replica.cc:333] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.511231 7478 raft_consensus.cc:2243] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.511651 7478 raft_consensus.cc:2272] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.513469 7478 ts_tablet_manager.cc:1916] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.518384 7736 raft_consensus.cc:493] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.518988 7736 raft_consensus.cc:515] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.522933 7736 leader_election.cc:290] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:58.523057 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:58.524623 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:18:58.526335 7264 leader_election.cc:343] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:58.528198 7261 leader_election.cc:343] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:58.529089 7261 leader_election.cc:304] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:58.530081 7736 raft_consensus.cc:2749] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:58.531433 7462 tablet_bootstrap.cc:492] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:58.551167 7478 ts_tablet_manager.cc:1929] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.551553 7478 log.cc:1199] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/56495c7cf3964743b96ab885afdaa152
I20260317 21:18:58.551748 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_674747b520f04b089691cd5513a58214"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.552215 7478 ts_tablet_manager.cc:1950] T 56495c7cf3964743b96ab885afdaa152 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.554229 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 100) took 1794 ms. Trace:
I20260317 21:18:58.554539 7478 rpcz_store.cc:276] 0317 21:18:56.759374 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.759480 (+ 106us) service_pool.cc:225] Handling call
0317 21:18:58.510434 (+1750954us) ts_tablet_manager.cc:1151] Deleting tablet 56495c7cf3964743b96ab885afdaa152
0317 21:18:58.510477 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.540912 (+ 30435us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.550932 (+ 10020us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.554167 (+ 3235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1750345}
I20260317 21:18:58.555697 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 56495c7cf3964743b96ab885afdaa152 (table default.Table_b1d7ee6663fe432a8940df97de65d99b [id=251402e78beb409192f3bd43ff21a9e6]) successfully deleted
I20260317 21:18:58.556180 7478 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 107) took 1564 ms. Trace:
I20260317 21:18:58.556486 7478 rpcz_store.cc:276] 0317 21:18:56.991962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:56.992141 (+ 179us) service_pool.cc:225] Handling call
0317 21:18:58.555814 (+1563673us) ts_tablet_manager.cc:1151] Deleting tablet 935faf7f9129440a91635bfa96191cd6
0317 21:18:58.555870 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:18:58.556128 (+ 258us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1563140}
I20260317 21:18:58.557693 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0a9cb20926ea45aaa419518e3f9c57ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.558099 7478 tablet_replica.cc:333] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.558744 7478 raft_consensus.cc:2243] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.559170 7478 raft_consensus.cc:2272] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.561017 7478 ts_tablet_manager.cc:1916] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.561393 7528 raft_consensus.cc:493] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.561862 7528 raft_consensus.cc:515] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.562590 7467 tablet_bootstrap.cc:492] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:58.563432 7467 ts_tablet_manager.cc:1403] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.176s user 0.010s sys 0.000s
I20260317 21:18:58.564338 7528 leader_election.cc:290] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.567782 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "163cc8c04ca94a269c7ae2efaa162525" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:58.568398 7183 raft_consensus.cc:2468] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
W20260317 21:18:58.569821 6973 leader_election.cc:343] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 163cc8c04ca94a269c7ae2efaa162525
I20260317 21:18:58.571614 6971 leader_election.cc:304] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:58.572930 7528 raft_consensus.cc:2804] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:58.573416 7528 raft_consensus.cc:493] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:58.573930 7528 raft_consensus.cc:3060] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.579142 7478 ts_tablet_manager.cc:1929] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.579653 7478 log.cc:1199] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/99b954ac41134300823458a7a25ef738
I20260317 21:18:58.580401 7478 ts_tablet_manager.cc:1950] T 99b954ac41134300823458a7a25ef738 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.582984 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 99b954ac41134300823458a7a25ef738 (table default.Table_93723f1459114d9ba82a21cee9158b3d [id=d079fd8ed7aa48ad9420c7cf7419c36e]) successfully deleted
I20260317 21:18:58.583181 7528 raft_consensus.cc:515] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:58.584766 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet d2f7c1058f8245428f38fe3245621b75 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d2f7c1058f8245428f38fe3245621b75 already in progress: creating tablet
I20260317 21:18:58.585811 7478 tablet_replica.cc:333] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.586019 7467 raft_consensus.cc:359] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.586759 7467 raft_consensus.cc:385] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.586155 7528 leader_election.cc:290] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.586969 7478 raft_consensus.cc:2243] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.587847 7478 raft_consensus.cc:2272] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.590150 7478 ts_tablet_manager.cc:1916] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.587033 7467 raft_consensus.cc:740] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.597376 7467 consensus_queue.cc:260] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.600466 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d50638a040344b81ad504d963feb3a4f" } new_table_name: "default.Table_0b009e9d95134391a1d2d41e80db86c7" modify_external_catalogs: true
I20260317 21:18:58.609319 7467 ts_tablet_manager.cc:1434] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.046s user 0.009s sys 0.000s
W20260317 21:18:58.616614 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2562cc2a3bb2430981c88475f9e1daa1 consensus state has no leader
I20260317 21:18:58.620405 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_aa8ccdaa054d482c934eb9db7cfaf5b0" } new_table_name: "default.Table_5b2a875dd16d4f49b1aaa18981060f82" modify_external_catalogs: true
I20260317 21:18:58.625630 7467 tablet_bootstrap.cc:492] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:58.628255 7462 tablet_bootstrap.cc:654] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.636201 7467 tablet_bootstrap.cc:654] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:18:58.637643 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a9cb20926ea45aaa419518e3f9c57ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:58.640771 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_674747b520f04b089691cd5513a58214 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:58.645896 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 221c71b908f942ca97f111f857f0335a has no consensus state
I20260317 21:18:58.648625 7478 ts_tablet_manager.cc:1929] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260317 21:18:58.649142 7478 log.cc:1199] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/019d90f6d70e46b087ea3c5ffd7cbd28
I20260317 21:18:58.650177 7478 ts_tablet_manager.cc:1950] T 019d90f6d70e46b087ea3c5ffd7cbd28 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.655701 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "163cc8c04ca94a269c7ae2efaa162525" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:58.656483 7183 raft_consensus.cc:3060] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.665452 7462 tablet_bootstrap.cc:492] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:58.665975 7462 ts_tablet_manager.cc:1403] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.135s user 0.011s sys 0.000s
W20260317 21:18:58.668900 6973 leader_election.cc:343] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 163cc8c04ca94a269c7ae2efaa162525
I20260317 21:18:58.670610 7183 raft_consensus.cc:2468] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:58.673271 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_d5a32bc495c44a738c19fc1679e42b10 [id=2ed77cec958e4b008989a60fb67a2adc]) successfully deleted
I20260317 21:18:58.676030 7462 raft_consensus.cc:359] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.686033 7462 raft_consensus.cc:385] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.686781 7462 raft_consensus.cc:740] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.690677 7459 tablet_bootstrap.cc:492] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:58.691605 7459 ts_tablet_manager.cc:1403] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.252s user 0.026s sys 0.020s
I20260317 21:18:58.696199 7459 raft_consensus.cc:359] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.697232 7459 raft_consensus.cc:385] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.697495 7459 raft_consensus.cc:740] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.698326 7459 consensus_queue.cc:260] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.675590 7467 tablet_bootstrap.cc:492] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:58.707877 7467 ts_tablet_manager.cc:1403] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.003s
I20260317 21:18:58.708474 6971 leader_election.cc:304] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:58.709978 7483 ts_tablet_manager.cc:1929] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.710378 7483 log.cc:1199] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/d2f7c1058f8245428f38fe3245621b75
I20260317 21:18:58.710613 7528 raft_consensus.cc:2804] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:58.710369 7467 raft_consensus.cc:359] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.711009 7467 raft_consensus.cc:385] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.711066 7483 ts_tablet_manager.cc:1950] T d2f7c1058f8245428f38fe3245621b75 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:58.711052 7528 raft_consensus.cc:697] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:58.711251 7467 raft_consensus.cc:740] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.711858 7528 consensus_queue.cc:237] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.713116 7459 ts_tablet_manager.cc:1434] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.021s user 0.006s sys 0.003s
I20260317 21:18:58.712952 7467 consensus_queue.cc:260] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.714424 6674 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 589aadc7897b422e90d3f7caed76ab59
I20260317 21:18:58.718932 7467 ts_tablet_manager.cc:1434] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260317 21:18:58.719712 7459 tablet_bootstrap.cc:492] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:58.722155 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 589aadc7897b422e90d3f7caed76ab59 with delete_type TABLET_DATA_DELETED (replaced by tablet 8744f6237e154dd99bc4c7e54822f7be) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:58.724444 7478 tablet_replica.cc:333] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:58.725633 7478 raft_consensus.cc:2243] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:58.726347 7478 raft_consensus.cc:2272] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.726787 6674 catalog_manager.cc:6741] ReplaceTablet: tablet 589aadc7897b422e90d3f7caed76ab59 deleted and replaced by tablet 8744f6237e154dd99bc4c7e54822f7be
I20260317 21:18:58.728113 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet 589aadc7897b422e90d3f7caed76ab59 with delete_type TABLET_DATA_DELETED (replaced by tablet 8744f6237e154dd99bc4c7e54822f7be) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:58.731091 7459 tablet_bootstrap.cc:654] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.731688 6674 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39038 (request call id 171) took 1167 ms. Trace:
I20260317 21:18:58.690346 7462 consensus_queue.cc:260] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.732748 6674 rpcz_store.cc:276] 0317 21:18:57.563810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.630908 (+ 67098us) service_pool.cc:225] Handling call
0317 21:18:57.636012 (+ 5104us) spinlock_profiling.cc:243] Waited 4.11 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b3bd1 00007fc05bef3227 00007fc05bef31b2 00007fc05bef30e1 00007fc05bef305c 00007fc05bef2312 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:58.730899 (+1094887us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:18:58.698516 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:18:58.698934 (+ 418us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:18:58.698958 (+ 24us) write_op.cc:435] Acquired schema lock
0317 21:18:58.698972 (+ 14us) tablet.cc:661] Decoding operations
0317 21:18:58.699391 (+ 419us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:18:58.699435 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0317 21:18:58.699466 (+ 31us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:18:58.699676 (+ 210us) tablet.cc:700] Row locks acquired
0317 21:18:58.699689 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:18:58.699839 (+ 150us) write_op.cc:270] Start()
0317 21:18:58.699926 (+ 87us) write_op.cc:276] Timestamp: P: 1773782338699750 usec, L: 0
0317 21:18:58.699940 (+ 14us) op_driver.cc:348] REPLICATION: starting
0317 21:18:58.700426 (+ 486us) log.cc:844] Serialized 625 byte log entry
0317 21:18:58.708190 (+ 7764us) op_driver.cc:464] REPLICATION: finished
0317 21:18:58.708601 (+ 411us) write_op.cc:301] APPLY: starting
0317 21:18:58.708681 (+ 80us) tablet.cc:1366] starting BulkCheckPresence
0317 21:18:58.708787 (+ 106us) tablet.cc:1369] finished BulkCheckPresence
0317 21:18:58.708900 (+ 113us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:18:58.709239 (+ 339us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:18:58.709436 (+ 197us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:18:58.709486 (+ 50us) write_op.cc:312] APPLY: finished
0317 21:18:58.709676 (+ 190us) log.cc:844] Serialized 29 byte log entry
0317 21:18:58.709835 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:58.709899 (+ 64us) write_op.cc:454] Released schema lock
0317 21:18:58.709929 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":9453824,"child_traces":[["op",{"apply.queue_time_us":261,"apply.run_cpu_time_us":1518,"apply.run_wall_time_us":1518,"num_ops":2,"prepare.queue_time_us":282,"raft.queue_time_us":101,"raft.run_cpu_time_us":759,"raft.run_wall_time_us":759,"replication_time_us":8150}]]}
I20260317 21:18:58.739418 7462 ts_tablet_manager.cc:1434] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.073s user 0.005s sys 0.000s
I20260317 21:18:58.740623 7462 tablet_bootstrap.cc:492] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:58.744894 7705 raft_consensus.cc:493] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.745383 7705 raft_consensus.cc:515] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.747350 7705 leader_election.cc:290] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.751696 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f980bd133da4b04b101e8df99ccca6f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:58.751624 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_71d3b46d9f544087abd87e1fe254c68a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.752270 7329 raft_consensus.cc:2468] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
W20260317 21:18:58.753265 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71d3b46d9f544087abd87e1fe254c68a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.753424 7483 tablet_replica.cc:333] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:58.753593 6973 leader_election.cc:304] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:58.754077 7483 raft_consensus.cc:2243] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:18:58.754388 7705 raft_consensus.cc:2804] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:58.754696 7705 raft_consensus.cc:493] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:58.755004 7705 raft_consensus.cc:3060] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.755015 7483 raft_consensus.cc:2272] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:58.757031 7478 ts_tablet_manager.cc:1916] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.758419 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet d2f7c1058f8245428f38fe3245621b75 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:18:58.767140 6672 catalog_manager.cc:5671] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 1 to 2, leader changed from c0ef92fef382401a9564c17e0c223f6a (127.6.122.66) to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 2 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:18:58.768544 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 06877bd7155842b9a5ae4a23ea68ada0 (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859]). Expected version 2 got 0
I20260317 21:18:58.769450 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet e1094068903840a8ab5d07c7fb0635bf (table default.Table_08bd63c86aca4345bc3ca2746d45456e [id=93623d5786f645c7a1ad496e656caa24]). Expected version 2 got 0
I20260317 21:18:58.771314 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0f98ce20efc14dfea9ce425147e744e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:58.772878 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f98ce20efc14dfea9ce425147e744e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.773666 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f980bd133da4b04b101e8df99ccca6f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:58.774310 7183 raft_consensus.cc:2468] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:58.777429 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c (DEFAULT_TABLE table=default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.778465 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 275e9cd7f14040d98eb85a6bb1010c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.719870 7467 tablet_bootstrap.cc:492] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:58.791222 7467 tablet_bootstrap.cc:654] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.794179 7483 ts_tablet_manager.cc:1916] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:58.794183 7705 raft_consensus.cc:515] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.797742 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 589aadc7897b422e90d3f7caed76ab59 with delete_type TABLET_DATA_DELETED (replaced by tablet 8744f6237e154dd99bc4c7e54822f7be) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:58.800781 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c (DEFAULT_TABLE table=default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.802263 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 275e9cd7f14040d98eb85a6bb1010c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.818027 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c (DEFAULT_TABLE table=default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.819564 7528 raft_consensus.cc:493] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.819883 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 275e9cd7f14040d98eb85a6bb1010c2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.819979 7528 raft_consensus.cc:515] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:58.819267 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 589aadc7897b422e90d3f7caed76ab59 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 589aadc7897b422e90d3f7caed76ab59 already in progress: creating tablet
I20260317 21:18:58.827196 7462 tablet_bootstrap.cc:654] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:58.830469 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 221c71b908f942ca97f111f857f0335a (DEFAULT_TABLE table=default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.831768 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221c71b908f942ca97f111f857f0335a. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:58.831800 6973 leader_election.cc:343] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 60f3d32e1f7649fa92861f51134cf000
I20260317 21:18:58.833228 7528 leader_election.cc:290] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.797781 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f980bd133da4b04b101e8df99ccca6f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:58.837587 7329 raft_consensus.cc:3060] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.844790 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 221c71b908f942ca97f111f857f0335a (DEFAULT_TABLE table=default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.845953 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221c71b908f942ca97f111f857f0335a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.856765 7705 leader_election.cc:290] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:58.856613 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:18:58.799027 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6f980bd133da4b04b101e8df99ccca6f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:58.857563 7183 raft_consensus.cc:3060] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:58.857715 6671 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 30) took 1068 ms (client timeout 1000 ms). Trace:
W20260317 21:18:58.858042 6671 rpcz_store.cc:269] 0317 21:18:57.789329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.789497 (+ 168us) service_pool.cc:225] Handling call
0317 21:18:57.799082 (+ 9585us) spinlock_profiling.cc:243] Waited 7.57 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:58.857634 (+1058552us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1,"spinlock_wait_cycles":17416448}
I20260317 21:18:58.858507 6667 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 174) took 1024 ms. Trace:
I20260317 21:18:58.824988 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60f3d32e1f7649fa92861f51134cf000" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:58.860939 7184 raft_consensus.cc:2468] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:18:58.861084 6669 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:18:58.834023 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 221c71b908f942ca97f111f857f0335a (DEFAULT_TABLE table=default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.862229 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 221c71b908f942ca97f111f857f0335a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.858865 6667 rpcz_store.cc:276] 0317 21:18:57.834108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.834236 (+ 128us) service_pool.cc:225] Handling call
0317 21:18:57.878567 (+ 44331us) spinlock_profiling.cc:243] Waited 43 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1386 00007fc05bed7024 00007fc05bed6fb2 00007fc05bed6ee1 00007fc05bed6e5c 00007fc05bed6112 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:57.879761 (+ 1194us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:18:57.881244 (+ 1483us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:18:57.881494 (+ 250us) catalog_manager.cc:2824] Locking tablets
0317 21:18:58.766417 (+884923us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:18:58.856174 (+ 89757us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:18:58.856382 (+ 208us) catalog_manager.cc:2872] Committing in-memory state
0317 21:18:58.856488 (+ 106us) catalog_manager.cc:2876] Aborting table tasks
0317 21:18:58.858438 (+ 1950us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:18:58.768750 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:18:58.769498 (+ 748us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:18:58.769521 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:18:58.769535 (+ 14us) tablet.cc:661] Decoding operations
0317 21:18:58.769829 (+ 294us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:18:58.769867 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0317 21:18:58.769887 (+ 20us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:18:58.770141 (+ 254us) tablet.cc:700] Row locks acquired
0317 21:18:58.770155 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:18:58.770264 (+ 109us) write_op.cc:270] Start()
0317 21:18:58.770343 (+ 79us) write_op.cc:276] Timestamp: P: 1773782338770220 usec, L: 0
0317 21:18:58.770357 (+ 14us) op_driver.cc:348] REPLICATION: starting
0317 21:18:58.770786 (+ 429us) log.cc:844] Serialized 728 byte log entry
0317 21:18:58.851180 (+ 80394us) op_driver.cc:464] REPLICATION: finished
0317 21:18:58.851792 (+ 612us) write_op.cc:301] APPLY: starting
0317 21:18:58.851845 (+ 53us) tablet.cc:1366] starting BulkCheckPresence
0317 21:18:58.851959 (+ 114us) tablet.cc:1369] finished BulkCheckPresence
0317 21:18:58.852051 (+ 92us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:18:58.852293 (+ 242us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:18:58.852438 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:18:58.852482 (+ 44us) write_op.cc:312] APPLY: finished
0317 21:18:58.852684 (+ 202us) log.cc:844] Serialized 29 byte log entry
0317 21:18:58.852821 (+ 137us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:58.852885 (+ 64us) write_op.cc:454] Released schema lock
0317 21:18:58.852926 (+ 41us) write_op.cc:341] FINISH: Updating metrics
0317 21:18:58.853153 (+ 227us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:58.853177 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":98794752,"child_traces":[["op",{"apply.queue_time_us":479,"apply.run_cpu_time_us":1330,"apply.run_wall_time_us":1330,"num_ops":2,"prepare.queue_time_us":237,"prepare.run_cpu_time_us":1907,"prepare.run_wall_time_us":2251,"replication_time_us":80738}]]}
I20260317 21:18:58.865438 6971 leader_election.cc:304] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:58.866703 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:57 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:58.867636 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:57 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:58.868866 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:57 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:58.872891 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet bcd99ca9f6c540e48e29d8659fbfe777 because the tablet was not found. No further retry: Not found: Tablet not found: bcd99ca9f6c540e48e29d8659fbfe777
W20260317 21:18:58.873242 6669 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:45181 (request call id 42) took 1062 ms (client timeout 1000 ms). Trace:
W20260317 21:18:58.873605 6669 rpcz_store.cc:269] 0317 21:18:57.810918 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.811131 (+ 213us) service_pool.cc:225] Handling call
0317 21:18:57.875797 (+ 64666us) spinlock_profiling.cc:243] Waited 62.7 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:58.873175 (+997378us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":144180096}
I20260317 21:18:58.873970 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet be2c469c792047d88fcf2ff9d53b7832 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:51 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:58.873971 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:58.875041 7705 raft_consensus.cc:2804] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:58.875192 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 with delete_type TABLET_DATA_DELETED (replaced by tablet 323717acdee841aaa1795a3ca851b40f) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:58.875443 7705 raft_consensus.cc:493] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:58.875771 7705 raft_consensus.cc:3060] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:58.882238 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 (DEFAULT_TABLE table=default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.883427 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b066a6a5c864beb8da59bd97bacd309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.883483 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 (DEFAULT_TABLE table=default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.884405 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b066a6a5c864beb8da59bd97bacd309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.884837 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 (DEFAULT_TABLE table=default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.885941 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b066a6a5c864beb8da59bd97bacd309. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.895422 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_02290a1a580945fc935a328e9a9faa11" } new_table_name: "default.Table_0475d441c7644d2388e6717057473374" modify_external_catalogs: true
I20260317 21:18:58.897159 6671 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_87cbb52aa7e249f98d6eb6ed317d3d46" } new_table_name: "default.Table_409b88634b48453daacc460109b51caf" modify_external_catalogs: true
I20260317 21:18:58.898675 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 (DEFAULT_TABLE table=default.Table_1d14e25fb3ec49bfb39f46819cdc1d5b [id=8fbd9827525947d58d75c279e4b92b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.899936 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2f891e712341868d94bab1dd72a9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.901988 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d6c3cd8934f349da9f5b908ea58fbb31"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:58.904012 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d6c3cd8934f349da9f5b908ea58fbb31 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.911713 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 (DEFAULT_TABLE table=default.Table_1d14e25fb3ec49bfb39f46819cdc1d5b [id=8fbd9827525947d58d75c279e4b92b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.912777 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2f891e712341868d94bab1dd72a9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.914242 7462 tablet_bootstrap.cc:492] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:58.914623 7462 ts_tablet_manager.cc:1403] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.174s user 0.006s sys 0.003s
I20260317 21:18:58.914263 7156 tablet_service.cc:1511] Processing CreateTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 (DEFAULT_TABLE table=default.Table_1d14e25fb3ec49bfb39f46819cdc1d5b [id=8fbd9827525947d58d75c279e4b92b6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.915587 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3e2f891e712341868d94bab1dd72a9e1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.918560 7459 tablet_bootstrap.cc:492] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:58.918982 7459 ts_tablet_manager.cc:1403] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.200s user 0.010s sys 0.000s
I20260317 21:18:58.921350 7459 raft_consensus.cc:359] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.921968 7459 raft_consensus.cc:385] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.922227 7459 raft_consensus.cc:740] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.922797 7459 consensus_queue.cc:260] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.928186 7459 ts_tablet_manager.cc:1434] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260317 21:18:58.929984 7008 tablet_service.cc:1511] Processing CreateTablet for tablet a662b365ec4846bf9e9ee243d41cb91a (DEFAULT_TABLE table=default.Table_351b690e266140a3a403e918f71f9d69 [id=ab3626e193c84f348f4cade48709a58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.931094 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a662b365ec4846bf9e9ee243d41cb91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.932152 7459 tablet_bootstrap.cc:492] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:18:58.939601 7183 raft_consensus.cc:2468] T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:58.943032 7462 raft_consensus.cc:359] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.943704 7462 raft_consensus.cc:385] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:58.943972 7462 raft_consensus.cc:740] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:58.944717 7462 consensus_queue.cc:260] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.953050 7462 ts_tablet_manager.cc:1434] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260317 21:18:58.943405 7705 raft_consensus.cc:515] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.942175 7155 tablet_service.cc:1511] Processing CreateTablet for tablet a662b365ec4846bf9e9ee243d41cb91a (DEFAULT_TABLE table=default.Table_351b690e266140a3a403e918f71f9d69 [id=ab3626e193c84f348f4cade48709a58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.943271 7717 raft_consensus.cc:493] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:58.977392 7717 raft_consensus.cc:515] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:58.981072 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a662b365ec4846bf9e9ee243d41cb91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.981912 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_10b2ba08717843c39931baab0bd22525"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:58.934221 7478 ts_tablet_manager.cc:1929] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:58.968596 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "60f3d32e1f7649fa92861f51134cf000" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:18:58.982903 7184 raft_consensus.cc:3060] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:18:58.983471 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10b2ba08717843c39931baab0bd22525 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:58.986855 7478 log.cc:1199] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/589aadc7897b422e90d3f7caed76ab59
I20260317 21:18:58.988057 7478 ts_tablet_manager.cc:1950] T 589aadc7897b422e90d3f7caed76ab59 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:58.994345 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 86ae75c4ce214954beecfab9275d0ea9 (DEFAULT_TABLE table=default.Table_05b675a3f5f649f0b419a59478fd907a [id=242dcdafc2b74e1ca17839f96e45af88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.995983 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ae75c4ce214954beecfab9275d0ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.959344 7300 tablet_service.cc:1511] Processing CreateTablet for tablet a662b365ec4846bf9e9ee243d41cb91a (DEFAULT_TABLE table=default.Table_351b690e266140a3a403e918f71f9d69 [id=ab3626e193c84f348f4cade48709a58d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:58.997362 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a662b365ec4846bf9e9ee243d41cb91a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.006377 7462 tablet_bootstrap.cc:492] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:59.008922 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 86ae75c4ce214954beecfab9275d0ea9 (DEFAULT_TABLE table=default.Table_05b675a3f5f649f0b419a59478fd907a [id=242dcdafc2b74e1ca17839f96e45af88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.010242 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ae75c4ce214954beecfab9275d0ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:58.972798 7329 raft_consensus.cc:2468] T 6f980bd133da4b04b101e8df99ccca6f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:58.975469 7483 ts_tablet_manager.cc:1929] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.4
I20260317 21:18:59.022055 7483 log.cc:1199] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/019d90f6d70e46b087ea3c5ffd7cbd28
I20260317 21:18:59.022902 7483 ts_tablet_manager.cc:1950] T 019d90f6d70e46b087ea3c5ffd7cbd28 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:59.025228 7483 tablet_replica.cc:333] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:59.025022 6973 leader_election.cc:304] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:18:59.025942 7528 raft_consensus.cc:2804] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:59.026468 7483 raft_consensus.cc:2243] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:59.026953 7483 raft_consensus.cc:2272] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:59.027118 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 019d90f6d70e46b087ea3c5ffd7cbd28 (table default.Table_d5a32bc495c44a738c19fc1679e42b10 [id=2ed77cec958e4b008989a60fb67a2adc]) successfully deleted
I20260317 21:18:59.034211 7462 tablet_bootstrap.cc:654] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:59.042277 7717 leader_election.cc:290] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:59.043007 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 86ae75c4ce214954beecfab9275d0ea9 (DEFAULT_TABLE table=default.Table_05b675a3f5f649f0b419a59478fd907a [id=242dcdafc2b74e1ca17839f96e45af88]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.044450 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 86ae75c4ce214954beecfab9275d0ea9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.046751 7459 tablet_bootstrap.cc:654] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:59.055789 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff (DEFAULT_TABLE table=default.Table_aef4da11f840465eb7fc80e6423d7454 [id=91eadfdf7d094f9fa83003dace842023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.057251 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5314c562bc9d457a8e1f4930e8d3a7ff. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:59.063160 6973 leader_election.cc:343] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 60f3d32e1f7649fa92861f51134cf000
I20260317 21:18:59.064112 7705 leader_election.cc:290] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:59.064325 7467 tablet_bootstrap.cc:492] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:18:59.064872 7467 ts_tablet_manager.cc:1403] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.345s user 0.010s sys 0.002s
I20260317 21:18:59.043118 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:59.065415 7329 raft_consensus.cc:2468] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:18:59.065922 7483 ts_tablet_manager.cc:1916] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:59.066636 7119 leader_election.cc:304] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:59.075354 7717 raft_consensus.cc:2804] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:18:59.075769 7717 raft_consensus.cc:493] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:18:59.076110 7717 raft_consensus.cc:3060] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:59.076192 7528 raft_consensus.cc:697] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:59.079490 7157 tablet_service.cc:1511] Processing CreateTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff (DEFAULT_TABLE table=default.Table_aef4da11f840465eb7fc80e6423d7454 [id=91eadfdf7d094f9fa83003dace842023]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.080844 7157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5314c562bc9d457a8e1f4930e8d3a7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.083299 7736 raft_consensus.cc:493] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.083767 7736 raft_consensus.cc:515] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.085747 7736 leader_election.cc:290] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:59.087936 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:59.089493 7261 leader_election.cc:343] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.089891 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 589aadc7897b422e90d3f7caed76ab59 (table default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]) successfully deleted
W20260317 21:18:59.091243 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet bcd99ca9f6c540e48e29d8659fbfe777 because the tablet was not found. No further retry: Not found: Tablet not found: bcd99ca9f6c540e48e29d8659fbfe777
W20260317 21:18:59.092201 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet be2c469c792047d88fcf2ff9d53b7832 because the tablet was not found. No further retry: Not found: Tablet not found: be2c469c792047d88fcf2ff9d53b7832
W20260317 21:18:59.093350 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet c1e672ca556e400ab05763abcd49ad77 because the tablet was not found. No further retry: Not found: Tablet not found: c1e672ca556e400ab05763abcd49ad77
I20260317 21:18:59.095264 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff (DEFAULT_TABLE table=default.Table_aef4da11f840465eb7fc80e6423d7454 [id=91eadfdf7d094f9fa83003dace842023]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:59.105208 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 8ba0874a8cd746e9a33596ea252b3b14 because the tablet was not found. No further retry: Not found: Tablet not found: 8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:18:59.106974 7580 consensus_queue.cc:1048] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:18:59.108945 7184 raft_consensus.cc:2468] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:18:59.111145 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5314c562bc9d457a8e1f4930e8d3a7ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.114511 7164 tablet_service.cc:1511] Processing CreateTablet for tablet a37e53e326914575830e30915ad69cc7 (DEFAULT_TABLE table=default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.115784 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37e53e326914575830e30915ad69cc7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:59.118507 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 35023195b9094d58827921a1dcb2ff05 consensus state has no leader
I20260317 21:18:59.119340 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet 78e2e20d320547b3bb1e03923de52822 from {username='slave'} at 127.0.0.1:39038
W20260317 21:18:59.121040 6671 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a221dfad830428db2329d6e543f41dc has no consensus state
I20260317 21:18:59.121752 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 06877bd7155842b9a5ae4a23ea68ada0
I20260317 21:18:59.123126 6670 catalog_manager.cc:6741] ReplaceTablet: tablet 06877bd7155842b9a5ae4a23ea68ada0 deleted and replaced by tablet c936edddda144fd7b0d6467cdc5ac5cc
I20260317 21:18:59.125067 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 with delete_type TABLET_DATA_DELETED (replaced by tablet c936edddda144fd7b0d6467cdc5ac5cc) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:59.132256 7736 raft_consensus.cc:493] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.133457 7161 tablet_service.cc:1558] Processing DeleteTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 with delete_type TABLET_DATA_DELETED (replaced by tablet c936edddda144fd7b0d6467cdc5ac5cc) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:59.133950 7736 raft_consensus.cc:515] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.134562 7478 tablet_replica.cc:333] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:59.135260 7478 raft_consensus.cc:2243] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:59.135744 7478 raft_consensus.cc:2272] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:59.137158 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4b718b3df61e49f397680a56009f72ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:59.138072 7736 leader_election.cc:290] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:18:59.138645 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4b718b3df61e49f397680a56009f72ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.138545 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "935faf7f9129440a91635bfa96191cd6" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:59.139816 7261 leader_election.cc:343] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.139930 7309 tablet_service.cc:1511] Processing CreateTablet for tablet a37e53e326914575830e30915ad69cc7 (DEFAULT_TABLE table=default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.141206 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37e53e326914575830e30915ad69cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.142282 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1885223af9524a43b0d34db30e68e6f1" } modify_external_catalogs: true
I20260317 21:18:59.142922 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1885223af9524a43b0d34db30e68e6f1" } modify_external_catalogs: true
I20260317 21:18:59.147881 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "935faf7f9129440a91635bfa96191cd6" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:59.148408 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:59.148785 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:59.150625 7161 tablet_service.cc:1511] Processing CreateTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 (DEFAULT_TABLE table=default.Table_fcec6e585d6245ff828c8c0c98450d63 [id=d238f6cb466c49d6af5ae380e967b054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.151810 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd99ca9f6c540e48e29d8659fbfe777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.153077 7528 consensus_queue.cc:237] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.153882 7478 ts_tablet_manager.cc:1916] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:18:59.154500 7264 leader_election.cc:343] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:18:59.154937 7120 leader_election.cc:343] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.154868 7264 leader_election.cc:304] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:18:59.155786 7264 leader_election.cc:343] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.156226 7264 leader_election.cc:304] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:59.156798 7736 raft_consensus.cc:2749] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:59.157766 7736 raft_consensus.cc:2749] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:59.158960 7306 tablet_service.cc:1511] Processing CreateTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 (DEFAULT_TABLE table=default.Table_fcec6e585d6245ff828c8c0c98450d63 [id=d238f6cb466c49d6af5ae380e967b054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.159966 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd99ca9f6c540e48e29d8659fbfe777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.165217 7705 raft_consensus.cc:493] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.179028 7462 tablet_bootstrap.cc:492] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:59.179543 7462 ts_tablet_manager.cc:1403] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.173s user 0.006s sys 0.006s
I20260317 21:18:59.182308 7462 raft_consensus.cc:359] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.072005 7467 raft_consensus.cc:359] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.183321 7462 raft_consensus.cc:385] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:59.183643 7467 raft_consensus.cc:385] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:59.183940 7467 raft_consensus.cc:740] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:18:59.183776 7462 raft_consensus.cc:740] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:59.184679 7467 consensus_queue.cc:260] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.185556 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d67952ad8d9d481e802d8f175c118d03"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:59.187054 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d67952ad8d9d481e802d8f175c118d03 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.211938 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 3d40549347f8463797d6791136883371 (DEFAULT_TABLE table=default.Table_1c93ac734b4d4ff68afbd9c72d470944 [id=5939f7e719474a598b8428b8af9a5b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.213083 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40549347f8463797d6791136883371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.213600 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 3d40549347f8463797d6791136883371 (DEFAULT_TABLE table=default.Table_1c93ac734b4d4ff68afbd9c72d470944 [id=5939f7e719474a598b8428b8af9a5b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.230118 7159 tablet_service.cc:1511] Processing CreateTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 (DEFAULT_TABLE table=default.Table_1885223af9524a43b0d34db30e68e6f1 [id=1293ecb1a2c24e7e987c4c0ed2524a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.231381 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8567ae10464694a1128ba2eb1d8ee1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.233989 6666 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_24103dddfd444503aa58762e5d598637 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 2 got 0
I20260317 21:18:59.234282 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:18:59.236191 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 78e2e20d320547b3bb1e03923de52822
I20260317 21:18:59.237025 7467 ts_tablet_manager.cc:1434] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.167s user 0.005s sys 0.001s
I20260317 21:18:59.237560 6667 catalog_manager.cc:6741] ReplaceTablet: tablet 78e2e20d320547b3bb1e03923de52822 deleted and replaced by tablet 203f33157a0d4b82aab44da452aa46dd
I20260317 21:18:59.238008 7467 tablet_bootstrap.cc:492] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:18:59.244143 7160 tablet_service.cc:1558] Processing DeleteTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:59.245496 7160 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:59.247467 7308 tablet_service.cc:1511] Processing CreateTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 (DEFAULT_TABLE table=default.Table_1885223af9524a43b0d34db30e68e6f1 [id=1293ecb1a2c24e7e987c4c0ed2524a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.248458 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:59.185127 7462 consensus_queue.cc:260] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.253891 7736 raft_consensus.cc:493] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.254374 7736 raft_consensus.cc:515] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.258621 7736 leader_election.cc:290] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:59.258917 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:59.260260 7261 leader_election.cc:343] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.261986 6673 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:18:59.262611 7462 ts_tablet_manager.cc:1434] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.083s user 0.005s sys 0.000s
I20260317 21:18:59.166329 7705 raft_consensus.cc:515] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.261615 6971 leader_election.cc:304] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:18:59.268425 6668 catalog_manager.cc:5671] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
W20260317 21:18:59.269353 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 102933us
I20260317 21:18:59.269724 6668 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:18:59.271175 6675 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet c936edddda144fd7b0d6467cdc5ac5cc
W20260317 21:18:59.278018 6672 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 51) took 1543 ms (client timeout 1000 ms). Trace:
W20260317 21:18:59.278661 6672 rpcz_store.cc:269] 0317 21:18:57.734953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:57.735209 (+ 256us) service_pool.cc:225] Handling call
0317 21:18:57.739397 (+ 4188us) spinlock_profiling.cc:243] Waited 1.89 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:18:59.277945 (+1538548us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:18:58.771793 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:18:58.772169 (+ 376us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:18:58.772191 (+ 22us) write_op.cc:435] Acquired schema lock
0317 21:18:58.772204 (+ 13us) tablet.cc:661] Decoding operations
0317 21:18:58.772425 (+ 221us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:18:58.772457 (+ 32us) write_op.cc:641] Partition lock acquired for write op
0317 21:18:58.772477 (+ 20us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:18:58.772602 (+ 125us) tablet.cc:700] Row locks acquired
0317 21:18:58.772613 (+ 11us) write_op.cc:260] PREPARE: finished
0317 21:18:58.772848 (+ 235us) write_op.cc:270] Start()
0317 21:18:58.772933 (+ 85us) write_op.cc:276] Timestamp: P: 1773782338772803 usec, L: 0
0317 21:18:58.772949 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:18:58.773398 (+ 449us) log.cc:844] Serialized 530 byte log entry
0317 21:18:58.851380 (+ 77982us) op_driver.cc:464] REPLICATION: finished
0317 21:18:58.853309 (+ 1929us) write_op.cc:301] APPLY: starting
0317 21:18:58.853357 (+ 48us) tablet.cc:1366] starting BulkCheckPresence
0317 21:18:58.853449 (+ 92us) tablet.cc:1369] finished BulkCheckPresence
0317 21:18:58.853547 (+ 98us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:18:58.853714 (+ 167us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:18:58.853847 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:18:58.853891 (+ 44us) write_op.cc:312] APPLY: finished
0317 21:18:58.854067 (+ 176us) log.cc:844] Serialized 23 byte log entry
0317 21:18:58.854203 (+ 136us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:18:58.854260 (+ 57us) write_op.cc:454] Released schema lock
0317 21:18:58.854290 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":7,"spinlock_wait_cycles":4343680,"child_traces":[["op",{"apply.queue_time_us":1811,"apply.run_cpu_time_us":1232,"apply.run_wall_time_us":18406,"num_ops":1,"prepare.queue_time_us":533,"prepare.run_cpu_time_us":1716,"prepare.run_wall_time_us":1809,"replication_time_us":78359,"thread_start_us":29689,"threads_started":1}]]}
I20260317 21:18:59.292879 7780 raft_consensus.cc:493] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.293493 7780 raft_consensus.cc:515] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.295971 7780 leader_election.cc:290] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:59.296761 7462 tablet_bootstrap.cc:492] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:59.312093 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40549347f8463797d6791136883371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.313663 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8567ae10464694a1128ba2eb1d8ee1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:59.327358 7119 leader_election.cc:343] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: c6d5385171364a078d968bb0a489383a
I20260317 21:18:59.330567 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ed089cf2f5554ed38d669ac4976ee78c" } new_table_name: "default.Table_1011d346d05146d7bb21b4e71f283d3c" modify_external_catalogs: true
I20260317 21:18:59.337445 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f" } new_table_name: "default.Table_44edff94b9794d2d806c2219236b3a8d" modify_external_catalogs: true
I20260317 21:18:59.337685 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f2bf8c9535e94059a16a46b599e39416"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:59.338711 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:18:59.339128 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f2bf8c9535e94059a16a46b599e39416 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.345909 7483 ts_tablet_manager.cc:1929] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:59.346345 7483 log.cc:1199] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/589aadc7897b422e90d3f7caed76ab59
I20260317 21:18:59.347011 7483 ts_tablet_manager.cc:1950] T 589aadc7897b422e90d3f7caed76ab59 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:59.349661 7483 tablet_replica.cc:333] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:59.369611 7013 tablet_service.cc:1511] Processing CreateTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 (DEFAULT_TABLE table=default.Table_fcec6e585d6245ff828c8c0c98450d63 [id=d238f6cb466c49d6af5ae380e967b054]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.369777 7478 ts_tablet_manager.cc:1929] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260317 21:18:59.371032 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bcd99ca9f6c540e48e29d8659fbfe777. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.372445 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 589aadc7897b422e90d3f7caed76ab59 (table default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]) successfully deleted
I20260317 21:18:59.372028 7478 log.cc:1199] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/06877bd7155842b9a5ae4a23ea68ada0
I20260317 21:18:59.378062 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 3d40549347f8463797d6791136883371 (DEFAULT_TABLE table=default.Table_1c93ac734b4d4ff68afbd9c72d470944 [id=5939f7e719474a598b8428b8af9a5b8b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:59.377856 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet bcd99ca9f6c540e48e29d8659fbfe777 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet bcd99ca9f6c540e48e29d8659fbfe777 already in progress: creating tablet
I20260317 21:18:59.379751 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d40549347f8463797d6791136883371. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.384622 7478 ts_tablet_manager.cc:1950] T 06877bd7155842b9a5ae4a23ea68ada0 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:18:59.391995 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (DEFAULT_TABLE table=default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.393095 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.401384 7462 tablet_bootstrap.cc:654] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:59.405668 7775 raft_consensus.cc:2804] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:59.397806 7483 raft_consensus.cc:2243] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:59.407902 7705 leader_election.cc:290] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:59.407650 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_484f7694e0d143c6b75c6403508c9fe8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:59.409329 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_484f7694e0d143c6b75c6403508c9fe8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.411077 7035 raft_consensus.cc:3060] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:59.406443 7483 raft_consensus.cc:2272] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:59.416251 7483 ts_tablet_manager.cc:1916] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:59.424678 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (DEFAULT_TABLE table=default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f [id=bcbc5e44c4ce46d2b795f6997fa466ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.425486 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:18:59.426244 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd3e35e25bf444cbd821ebd9f5ee018. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:59.430771 6973 consensus_peers.cc:597] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d59deea79cfb4a83831bb0227797c690. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:18:59.441494 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet fc8567ae10464694a1128ba2eb1d8ee1 because the tablet was not found. No further retry: Not found: Tablet not found: fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:18:59.442209 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_5661b77c4d044e2ab5f264577eb585ce"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:59.442593 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 78e2e20d320547b3bb1e03923de52822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e2e20d320547b3bb1e03923de52822 already in progress: creating tablet
W20260317 21:18:59.443686 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5661b77c4d044e2ab5f264577eb585ce in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.444412 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bdd5608a590847cf98a367c051f62b45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:59.445339 7779 consensus_queue.cc:1048] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:18:59.445923 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bdd5608a590847cf98a367c051f62b45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.447770 7160 tablet_service.cc:1511] Processing CreateTablet for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (DEFAULT_TABLE table=default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.448889 7160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.454072 6674 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d67952ad8d9d481e802d8f175c118d03" } modify_external_catalogs: true
I20260317 21:18:59.454586 6674 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d67952ad8d9d481e802d8f175c118d03" } modify_external_catalogs: true
I20260317 21:18:59.456053 7462 tablet_bootstrap.cc:492] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:59.456452 7462 ts_tablet_manager.cc:1403] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.160s user 0.004s sys 0.010s
I20260317 21:18:59.458931 7462 raft_consensus.cc:359] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.463626 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 323717acdee841aaa1795a3ca851b40f (DEFAULT_TABLE table=default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.464792 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323717acdee841aaa1795a3ca851b40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.464689 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 57e721ac4b24438b9b2a0e2569027236 (DEFAULT_TABLE table=default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.465689 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e721ac4b24438b9b2a0e2569027236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.468474 7717 raft_consensus.cc:515] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.473479 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:18:59.474087 7329 raft_consensus.cc:3060] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:18:59.459589 7462 raft_consensus.cc:385] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:59.482970 7305 tablet_service.cc:1511] Processing CreateTablet for tablet c9b7f7c864004b9abde0a0201e196165 (DEFAULT_TABLE table=default.Table_ce47367ed77d4c06ad8c624c31dd3b2b [id=7f6c80768baf4e05aad94f1b86cfdfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.484143 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b7f7c864004b9abde0a0201e196165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.481413 7462 raft_consensus.cc:740] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:59.493275 7462 consensus_queue.cc:260] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:59.501794 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 2a221dfad830428db2329d6e543f41dc (table default.Table_409b88634b48453daacc460109b51caf [id=2d22abe029d348c484e6aff5df8b3d29], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a221dfad830428db2329d6e543f41dc has no consensus state
I20260317 21:18:59.506129 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f749358db0f74ef78d9eee23b72bfe3e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:18:59.511547 6666 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_08bd63c86aca4345bc3ca2746d45456e [id=93623d5786f645c7a1ad496e656caa24] alter complete (version 2)
W20260317 21:18:59.521201 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 63a27911e0e545f5811eb075997d0b0f consensus state has no leader
W20260317 21:18:59.522535 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fd3e35e25bf444cbd821ebd9f5ee018 consensus state has no leader
I20260317 21:18:59.531348 7307 tablet_service.cc:1511] Processing CreateTablet for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (DEFAULT_TABLE table=default.Table_bfc5d1e1d8fa42cf920a94eab51746b3 [id=3cfede0a76b448daab974b495b79aa67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.531790 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 935faf7f9129440a91635bfa96191cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:56 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:59.533270 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:58 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:59.533903 6666 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6d4e07837b8945f981be0ae07f5b4ea0" } modify_external_catalogs: true
I20260317 21:18:59.535272 6666 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6d4e07837b8945f981be0ae07f5b4ea0" } modify_external_catalogs: true
I20260317 21:18:59.539254 7163 tablet_service.cc:1511] Processing CreateTablet for tablet c9b7f7c864004b9abde0a0201e196165 (DEFAULT_TABLE table=default.Table_ce47367ed77d4c06ad8c624c31dd3b2b [id=7f6c80768baf4e05aad94f1b86cfdfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.540452 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b7f7c864004b9abde0a0201e196165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.544543 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd1d8d70e9849fa9e107b2dd3355c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.550748 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 564d41fc13b04ec688fc38a78f55e83d (DEFAULT_TABLE table=default.Table_cf3bed4cf36c4e05a6ca6ab34245a067 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.551860 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564d41fc13b04ec688fc38a78f55e83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.554436 7156 tablet_service.cc:1511] Processing CreateTablet for tablet 57e721ac4b24438b9b2a0e2569027236 (DEFAULT_TABLE table=default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.555837 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e721ac4b24438b9b2a0e2569027236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.560880 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 039d8bed256a43a6b98075b8430cbf3b (DEFAULT_TABLE table=default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.561530 7157 tablet_service.cc:1511] Processing CreateTablet for tablet 323717acdee841aaa1795a3ca851b40f (DEFAULT_TABLE table=default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.562094 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039d8bed256a43a6b98075b8430cbf3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.562548 6672 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:18:59.562712 7157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323717acdee841aaa1795a3ca851b40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.563184 6672 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]). Expected version 1 got 0
W20260317 21:18:59.564263 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 134184us
I20260317 21:18:59.464514 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 with delete_type TABLET_DATA_DELETED (replaced by tablet c936edddda144fd7b0d6467cdc5ac5cc) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.474143 7008 tablet_service.cc:1511] Processing CreateTablet for tablet a37e53e326914575830e30915ad69cc7 (DEFAULT_TABLE table=default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.481339 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.482667 7012 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.483808 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet c1e672ca556e400ab05763abcd49ad77 with delete_type TABLET_DATA_DELETED (replaced by tablet dc929bae186841d2aba674659e457301) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.485968 7011 tablet_service.cc:1558] Processing DeleteTablet for tablet d59deea79cfb4a83831bb0227797c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.572561 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a37e53e326914575830e30915ad69cc7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.493726 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:59.494933 7775 raft_consensus.cc:697] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:18:59.581375 7775 consensus_queue.cc:237] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.588554 7299 tablet_service.cc:1558] Processing DeleteTablet for tablet dc929bae186841d2aba674659e457301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:59.590641 6971 leader_election.cc:304] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
W20260317 21:18:59.591962 6971 consensus_peers.cc:597] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06877bd7155842b9a5ae4a23ea68ada0. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:59.599076 7791 tablet_replica.cc:333] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:18:59.600104 7483 ts_tablet_manager.cc:1929] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260317 21:18:59.601073 7717 leader_election.cc:290] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:59.606575 6672 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:18:59.613679 7462 ts_tablet_manager.cc:1434] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.157s user 0.004s sys 0.001s
I20260317 21:18:59.626869 6674 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 29442c09e8074ac3b223320819399677
I20260317 21:18:59.627668 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (DEFAULT_TABLE table=default.Table_bfc5d1e1d8fa42cf920a94eab51746b3 [id=3cfede0a76b448daab974b495b79aa67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.629022 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd1d8d70e9849fa9e107b2dd3355c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.629246 7462 tablet_bootstrap.cc:492] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:59.535841 7014 tablet_service.cc:1511] Processing CreateTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 (DEFAULT_TABLE table=default.Table_1885223af9524a43b0d34db30e68e6f1 [id=1293ecb1a2c24e7e987c4c0ed2524a94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.602028 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 8ba0874a8cd746e9a33596ea252b3b14 with delete_type TABLET_DATA_DELETED (replaced by tablet 323717acdee841aaa1795a3ca851b40f) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.605319 7011 tablet_service.cc:1511] Processing CreateTablet for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (DEFAULT_TABLE table=default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.602144 7528 raft_consensus.cc:2749] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:59.607609 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (DEFAULT_TABLE table=default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f [id=bcbc5e44c4ce46d2b795f6997fa466ce]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.609647 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 323717acdee841aaa1795a3ca851b40f (DEFAULT_TABLE table=default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.611606 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 57e721ac4b24438b9b2a0e2569027236 (DEFAULT_TABLE table=default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.654618 7011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.613911 7017 tablet_service.cc:1511] Processing CreateTablet for tablet c9b7f7c864004b9abde0a0201e196165 (DEFAULT_TABLE table=default.Table_ce47367ed77d4c06ad8c624c31dd3b2b [id=7f6c80768baf4e05aad94f1b86cfdfdd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.655864 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c9b7f7c864004b9abde0a0201e196165. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.660956 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 323717acdee841aaa1795a3ca851b40f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.665834 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fc8567ae10464694a1128ba2eb1d8ee1. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:18:59.666882 7264 leader_election.cc:343] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.667402 7264 leader_election.cc:304] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:18:59.678457 6674 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]). Expected version 1 got 0
I20260317 21:18:59.638655 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6d5385171364a078d968bb0a489383a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:18:59.681830 7785 raft_consensus.cc:2749] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:59.682789 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57e721ac4b24438b9b2a0e2569027236. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.533803 7459 tablet_bootstrap.cc:492] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:18:59.684873 7459 ts_tablet_manager.cc:1403] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.753s user 0.010s sys 0.014s
I20260317 21:18:59.688963 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd3e35e25bf444cbd821ebd9f5ee018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.689826 7329 raft_consensus.cc:2468] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:18:59.690237 7791 raft_consensus.cc:2243] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 LEADER]: Raft consensus shutting down.
I20260317 21:18:59.691581 7119 leader_election.cc:304] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:18:59.693596 7717 raft_consensus.cc:2804] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:18:59.694555 7717 raft_consensus.cc:697] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:18:59.695899 7462 tablet_bootstrap.cc:654] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:59.703881 7153 tablet_service.cc:1558] Processing DeleteTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:59.705397 7478 tablet_replica.cc:333] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:18:59.600512 7483 log.cc:1199] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/06877bd7155842b9a5ae4a23ea68ada0
I20260317 21:18:59.723330 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_17cf4ddfbd3d47ba99971456fc3873f6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:59.724889 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17cf4ddfbd3d47ba99971456fc3873f6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.731986 7483 ts_tablet_manager.cc:1950] T 06877bd7155842b9a5ae4a23ea68ada0 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:59.746939 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1e2eb35c075a45d1afe94750072f9341" } new_table_name: "default.Table_66c25b9b83724e718f148f260ec5f672" modify_external_catalogs: true
I20260317 21:18:59.747052 6671 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1d14e25fb3ec49bfb39f46819cdc1d5b" } new_table_name: "default.Table_0c31d9e5b0bd4454acccb920047a5e73" modify_external_catalogs: true
W20260317 21:18:59.762683 7164 consensus_meta.cc:287] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a: Time spent flushing consensus metadata: real 0.570s user 0.008s sys 0.000s
I20260317 21:18:59.765007 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (DEFAULT_TABLE table=default.Table_bcdedd2686d74b88b97e7a0dfa9ad17f [id=bcbc5e44c4ce46d2b795f6997fa466ce]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:18:59.768885 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 78e2e20d320547b3bb1e03923de52822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e2e20d320547b3bb1e03923de52822 already in progress: creating tablet
W20260317 21:18:50.463111 7504 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:18:59.769671 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet fc8567ae10464694a1128ba2eb1d8ee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc8567ae10464694a1128ba2eb1d8ee1 already in progress: creating tablet
W20260317 21:18:51.755894 7496 debug-util.cc:398] Leaking SignalData structure 0x7b080002eda0 after lost signal to thread 7443
I20260317 21:18:59.770462 7483 tablet_replica.cc:333] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
W20260317 21:18:59.763622 7443 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 9.312s user 0.744s sys 1.246s
I20260317 21:18:59.771129 7483 raft_consensus.cc:2243] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260317 21:18:59.771406 7443 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 9.320s user 0.744s sys 1.246s
I20260317 21:18:59.779130 7717 consensus_queue.cc:237] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:18:59.782550 7796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:18:59.678810 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 564d41fc13b04ec688fc38a78f55e83d (DEFAULT_TABLE table=default.Table_cf3bed4cf36c4e05a6ca6ab34245a067 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.793777 7443 server_base.cc:1060] running on GCE node
I20260317 21:18:59.709797 7528 consensus_queue.cc:1048] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:18:59.795517 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260317 21:18:59.797369 6971 consensus_peers.cc:597] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:59.755702 7786 raft_consensus.cc:493] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.798507 7483 raft_consensus.cc:2272] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:59.798123 7786 raft_consensus.cc:515] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.800283 7483 ts_tablet_manager.cc:1916] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:59.802023 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fb7cb082cf47ce8cf2a98146633af7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:59.803926 6971 leader_election.cc:343] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.805300 7786 leader_election.cc:290] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:18:59.807169 7791 raft_consensus.cc:2272] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:59.810308 7791 ts_tablet_manager.cc:1916] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:59.811506 7443 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:18:59.812219 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 60f3d32e1f7649fa92861f51134cf000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:18:59.815551 7443 hybrid_clock.cc:648] HybridClock initialized: now 1773782339815505 us; error 55 us; skew 500 ppm
I20260317 21:18:59.816918 7443 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:18:59.817996 7785 raft_consensus.cc:493] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:18:59.818437 7785 raft_consensus.cc:515] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.820333 7785 leader_election.cc:290] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:18:59.821573 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc929bae186841d2aba674659e457301" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:18:59.823202 7261 leader_election.cc:343] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:18:59.834023 6669 master_service.cc:954] ReplaceTablet: received request to replace tablet 478f55d4c3034be0b1ef336a173c55b2 from {username='slave'} at 127.0.0.1:39038
I20260317 21:18:59.838596 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:18:59.838835 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_c82fde1f1c4e4ff4ab059d998ab8399a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:59.840180 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.840543 6673 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_212ed8405de344519eaecc32555fcbb5" } modify_external_catalogs: true
I20260317 21:18:59.837580 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f95f849f2a4340bbb5f515633e6f50fe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:18:59.840183 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c82fde1f1c4e4ff4ab059d998ab8399a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:18:59.842151 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f95f849f2a4340bbb5f515633e6f50fe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:18:59.844878 6673 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_212ed8405de344519eaecc32555fcbb5" } modify_external_catalogs: true
I20260317 21:18:59.848773 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 59fce28afbeb4d8eadf63f9a97d1183d
I20260317 21:18:59.851699 7462 tablet_bootstrap.cc:492] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:18:59.852342 7462 ts_tablet_manager.cc:1403] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.223s user 0.018s sys 0.009s
I20260317 21:18:59.852584 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:18:59.854847 7462 raft_consensus.cc:359] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.873001 7443 webserver.cc:492] Webserver started at http://127.6.122.124:40359/ using document root <none> and password file <none>
I20260317 21:18:59.874703 7443 fs_manager.cc:362] Metadata directory not provided
I20260317 21:18:59.875094 7443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:18:59.882772 7462 raft_consensus.cc:385] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:18:59.884783 7462 raft_consensus.cc:740] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:18:59.886898 7462 consensus_queue.cc:260] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.894953 7467 tablet_bootstrap.cc:654] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:59.907191 7462 ts_tablet_manager.cc:1434] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.054s user 0.005s sys 0.000s
I20260317 21:18:59.910248 7462 tablet_bootstrap.cc:492] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:18:59.935784 7462 tablet_bootstrap.cc:654] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:18:59.939149 7443 fs_manager.cc:714] Time spent opening directory manager: real 0.048s user 0.013s sys 0.024s
I20260317 21:18:59.854475 7018 tablet_service.cc:1511] Processing CreateTablet for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (DEFAULT_TABLE table=default.Table_bfc5d1e1d8fa42cf920a94eab51746b3 [id=3cfede0a76b448daab974b495b79aa67]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.953756 7018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2fd1d8d70e9849fa9e107b2dd3355c58. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.856540 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet 53dd9f8a1ccc47f884bfba99781aed16 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.858269 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 564d41fc13b04ec688fc38a78f55e83d (DEFAULT_TABLE table=default.Table_cf3bed4cf36c4e05a6ca6ab34245a067 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.860284 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 039d8bed256a43a6b98075b8430cbf3b (DEFAULT_TABLE table=default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:18:59.958148 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039d8bed256a43a6b98075b8430cbf3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.861865 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.868245 7017 tablet_service.cc:1558] Processing DeleteTablet for tablet 163cc8c04ca94a269c7ae2efaa162525 with delete_type TABLET_DATA_DELETED (replaced by tablet 9b6275af1c9c41ae8e4515deabdbdee1) from {username='slave'} at 127.0.0.1:47442
I20260317 21:18:59.961817 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564d41fc13b04ec688fc38a78f55e83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:18:59.791356 7459 raft_consensus.cc:359] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:18:59.945308 7483 ts_tablet_manager.cc:1929] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:18:59.989375 7483 log.cc:1199] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/935faf7f9129440a91635bfa96191cd6
I20260317 21:18:59.990060 7483 ts_tablet_manager.cc:1950] T 935faf7f9129440a91635bfa96191cd6 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:18:59.992537 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 935faf7f9129440a91635bfa96191cd6 (table default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]) successfully deleted
W20260317 21:18:59.993860 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 019d90f6d70e46b087ea3c5ffd7cbd28 because the tablet was not found. No further retry: Not found: Tablet not found: 019d90f6d70e46b087ea3c5ffd7cbd28
W20260317 21:18:59.815065 6973 leader_election.cc:343] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:18:59.994688 6973 leader_election.cc:304] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
W20260317 21:18:59.995502 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 180470us
I20260317 21:18:59.996338 7786 raft_consensus.cc:2749] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:18:59.992607 7483 tablet_replica.cc:333] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:18:59.997586 7483 raft_consensus.cc:2243] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:18:59.998024 7483 raft_consensus.cc:2272] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:18:59.999667 7483 ts_tablet_manager.cc:1916] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:18:59.999326 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc929bae186841d2aba674659e457301" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:00.002341 7264 leader_election.cc:343] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.002997 7264 leader_election.cc:304] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:00.003540 7264 raft_consensus.cc:2686] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260317 21:19:00.006386 6973 consensus_peers.cc:597] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 163cc8c04ca94a269c7ae2efaa162525. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:00.011878 7802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:00.012439 7459 raft_consensus.cc:385] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.013104 7459 raft_consensus.cc:740] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.013901 7459 consensus_queue.cc:260] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.017866 7459 ts_tablet_manager.cc:1434] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.333s user 0.006s sys 0.001s
I20260317 21:19:00.021552 7459 tablet_bootstrap.cc:492] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:18:59.989403 6971 consensus_peers.cc:597] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d59deea79cfb4a83831bb0227797c690. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:18:59.931427 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 60f3d32e1f7649fa92861f51134cf000 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:52 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:00.031586 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fd3e35e25bf444cbd821ebd9f5ee018. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.033773 7478 raft_consensus.cc:2243] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:00.034430 7478 raft_consensus.cc:2272] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:00.038570 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6caf05c79a5f470e8b7796d0e85e36f1 consensus state has no leader
W20260317 21:19:00.040318 6671 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3e2f891e712341868d94bab1dd72a9e1 consensus state has no leader
I20260317 21:19:00.042757 7467 tablet_bootstrap.cc:492] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:00.043226 7467 ts_tablet_manager.cc:1403] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.805s user 0.008s sys 0.006s
I20260317 21:19:00.044219 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 564d41fc13b04ec688fc38a78f55e83d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.045758 7443 fs_manager.cc:730] Time spent opening block manager: real 0.103s user 0.002s sys 0.005s
I20260317 21:19:00.049324 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:58 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.046247 7443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
format_stamp: "Formatted at 2026-03-17 21:18:36 on dist-test-slave-979h"
I20260317 21:19:00.051134 7443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:19:00.056569 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 589aadc7897b422e90d3f7caed76ab59 with delete_type TABLET_DATA_DELETED (replaced by tablet 8744f6237e154dd99bc4c7e54822f7be) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.058384 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 039d8bed256a43a6b98075b8430cbf3b (DEFAULT_TABLE table=default.Table_f280d2394ccd432ea164002a2b8ce9f4 [id=6761934b095549bb86447782a7f7a611]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.059168 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0c31d9e5b0bd4454acccb920047a5e73" } modify_external_catalogs: true
I20260317 21:19:00.059515 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 039d8bed256a43a6b98075b8430cbf3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.059713 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0c31d9e5b0bd4454acccb920047a5e73" } modify_external_catalogs: true
I20260317 21:19:00.062165 7467 raft_consensus.cc:359] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.062899 7467 raft_consensus.cc:385] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.063211 7467 raft_consensus.cc:740] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.063932 7467 consensus_queue.cc:260] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.089540 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet c6d5385171364a078d968bb0a489383a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.090431 7153 tablet_service.cc:1558] Processing DeleteTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.098644 7467 ts_tablet_manager.cc:1434] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.055s user 0.006s sys 0.001s
I20260317 21:19:00.099581 7467 tablet_bootstrap.cc:492] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:00.111308 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet 35023195b9094d58827921a1dcb2ff05 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:00.121488 7483 ts_tablet_manager.cc:1929] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:00.126346 7483 log.cc:1199] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/dc929bae186841d2aba674659e457301
I20260317 21:19:00.127575 7483 ts_tablet_manager.cc:1950] T dc929bae186841d2aba674659e457301 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:00.127775 7467 tablet_bootstrap.cc:654] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.131296 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet dc929bae186841d2aba674659e457301 (table default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]) successfully deleted
I20260317 21:19:00.138417 7462 tablet_bootstrap.cc:492] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:00.138840 7462 ts_tablet_manager.cc:1403] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.229s user 0.010s sys 0.018s
I20260317 21:19:00.141167 7462 raft_consensus.cc:359] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.141816 7462 raft_consensus.cc:385] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.142064 7462 raft_consensus.cc:740] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.142654 7462 consensus_queue.cc:260] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.143369 7478 ts_tablet_manager.cc:1916] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:00.153982 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 59fce28afbeb4d8eadf63f9a97d1183d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fce28afbeb4d8eadf63f9a97d1183d already in progress: creating tablet
I20260317 21:19:00.156325 7462 ts_tablet_manager.cc:1434] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.017s user 0.006s sys 0.001s
I20260317 21:19:00.157428 7462 tablet_bootstrap.cc:492] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:00.136307 7791 ts_tablet_manager.cc:1929] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260317 21:19:00.159337 7791 log.cc:1199] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/06877bd7155842b9a5ae4a23ea68ada0
I20260317 21:19:00.163439 7462 tablet_bootstrap.cc:654] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.165156 7791 ts_tablet_manager.cc:1950] T 06877bd7155842b9a5ae4a23ea68ada0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
W20260317 21:19:00.183254 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet fc8567ae10464694a1128ba2eb1d8ee1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fc8567ae10464694a1128ba2eb1d8ee1 already in progress: creating tablet
I20260317 21:19:00.183539 7467 tablet_bootstrap.cc:492] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:00.183920 7467 ts_tablet_manager.cc:1403] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.085s user 0.009s sys 0.000s
I20260317 21:19:00.188220 7467 raft_consensus.cc:359] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.189219 7467 raft_consensus.cc:385] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.189553 7467 raft_consensus.cc:740] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
W20260317 21:19:00.190752 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 78e2e20d320547b3bb1e03923de52822 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78e2e20d320547b3bb1e03923de52822 already in progress: creating tablet
I20260317 21:19:00.190428 7467 consensus_queue.cc:260] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.191737 7791 tablet_replica.cc:333] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:00.203559 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:00.255766 7791 raft_consensus.cc:2243] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:00.256402 7791 raft_consensus.cc:2272] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.258872 7791 ts_tablet_manager.cc:1916] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.264838 7462 tablet_bootstrap.cc:492] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:00.265610 7462 ts_tablet_manager.cc:1403] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.108s user 0.036s sys 0.017s
I20260317 21:19:00.268232 7462 raft_consensus.cc:359] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.269109 7462 raft_consensus.cc:385] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.269461 7462 raft_consensus.cc:740] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.270229 7462 consensus_queue.cc:260] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:00.271134 7120 leader_election.cc:343] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.273717 7120 leader_election.cc:304] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:00.272229 7462 ts_tablet_manager.cc:1434] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260317 21:19:00.279363 7462 tablet_bootstrap.cc:492] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:00.287067 7467 ts_tablet_manager.cc:1434] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.103s user 0.003s sys 0.002s
I20260317 21:19:00.293090 7158 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:42830 (request call id 131) took 1088 ms. Trace:
I20260317 21:19:00.293565 7158 rpcz_store.cc:276] 0317 21:18:59.204760 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.206694 (+ 1934us) service_pool.cc:225] Handling call
0317 21:18:59.212565 (+ 5871us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 21:18:59.212690 (+ 125us) ts_tablet_manager.cc:682] Creating new metadata...
0317 21:18:59.820570 (+607880us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:00.240451 (+419881us) tablet_replica.cc:206] Creating consensus instance
0317 21:19:00.293025 (+ 52574us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:00.295413 7462 tablet_bootstrap.cc:654] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.305529 7717 raft_consensus.cc:2749] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:00.308511 7120 leader_election.cc:343] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.328222 7478 ts_tablet_manager.cc:1929] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:00.328891 7478 log.cc:1199] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/163cc8c04ca94a269c7ae2efaa162525
I20260317 21:19:00.329711 7478 ts_tablet_manager.cc:1950] T 163cc8c04ca94a269c7ae2efaa162525 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:00.334611 7459 tablet_bootstrap.cc:654] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.338985 7467 tablet_bootstrap.cc:492] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:00.347460 7467 tablet_bootstrap.cc:654] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.348070 7785 raft_consensus.cc:493] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.348529 7785 raft_consensus.cc:515] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.350875 7785 leader_election.cc:290] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.353962 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eee8b5588c24661889c593797c94269" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:00.355548 7264 leader_election.cc:343] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.362814 7785 raft_consensus.cc:493] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.363402 7785 raft_consensus.cc:515] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.366465 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb332f981fc84a178fadae2f4908bdde" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:00.368286 7264 leader_election.cc:343] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.370592 7785 leader_election.cc:290] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.372903 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb332f981fc84a178fadae2f4908bdde" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:00.373467 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eee8b5588c24661889c593797c94269" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:00.378283 7261 leader_election.cc:343] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.378748 7261 leader_election.cc:304] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:00.379884 7261 leader_election.cc:343] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.380265 7261 leader_election.cc:304] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:00.381919 7785 raft_consensus.cc:2749] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:00.382650 7785 raft_consensus.cc:2749] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:00.375710 7478 tablet_replica.cc:333] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:00.384060 7478 raft_consensus.cc:2243] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:00.385571 7478 raft_consensus.cc:2272] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.389842 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 163cc8c04ca94a269c7ae2efaa162525 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]) successfully deleted
I20260317 21:19:00.408869 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_674747b520f04b089691cd5513a58214" } new_table_name: "default.Table_d34a86e0f32b40099d38cb403d3b5fc8" modify_external_catalogs: true
I20260317 21:19:00.409386 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_cb3946746d5d4e008932e6f7f476a59a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:00.411651 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb3946746d5d4e008932e6f7f476a59a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:00.415787 7478 ts_tablet_manager.cc:1916] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.417382 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_8ca6f39801634869ae519a57be7cfd69" } new_table_name: "default.Table_40785d5a99c543259bad7c3c6bcbce39" modify_external_catalogs: true
I20260317 21:19:00.417845 7717 raft_consensus.cc:493] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.418293 7717 raft_consensus.cc:515] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.420421 7717 leader_election.cc:290] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:00.420965 6669 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:00.421934 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fb7cb082cf47ce8cf2a98146633af7" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:00.422525 7037 raft_consensus.cc:2468] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:00.423682 7120 leader_election.cc:304] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:00.425676 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a44a297079347bab6f8be6abe332bfa) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:00.426956 7119 leader_election.cc:343] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:19:00.428530 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a44a297079347bab6f8be6abe332bfa) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:00.428499 7717 raft_consensus.cc:2804] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:00.429148 7717 raft_consensus.cc:493] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:00.429474 7483 tablet_replica.cc:333] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:00.429867 7528 raft_consensus.cc:493] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.430480 7483 raft_consensus.cc:2243] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:00.430315 7528 raft_consensus.cc:515] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.431102 7483 raft_consensus.cc:2272] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.432173 7528 leader_election.cc:290] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:00.434013 6669 catalog_manager.cc:6741] ReplaceTablet: tablet 478f55d4c3034be0b1ef336a173c55b2 deleted and replaced by tablet 8a44a297079347bab6f8be6abe332bfa
W20260317 21:19:00.434777 6971 leader_election.cc:343] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 589aadc7897b422e90d3f7caed76ab59
I20260317 21:19:00.429512 7717 raft_consensus.cc:3060] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:00.436204 7483 ts_tablet_manager.cc:1916] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.441542 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a44a297079347bab6f8be6abe332bfa) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.445971 7791 ts_tablet_manager.cc:1929] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:00.449844 7717 raft_consensus.cc:515] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:00.464280 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 84425.3us
I20260317 21:19:00.469507 7462 tablet_bootstrap.cc:492] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:00.470098 7462 ts_tablet_manager.cc:1403] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.191s user 0.016s sys 0.029s
I20260317 21:19:00.472060 7467 tablet_bootstrap.cc:492] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:00.473805 7462 raft_consensus.cc:359] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.474781 7462 raft_consensus.cc:385] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.475291 7462 raft_consensus.cc:740] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.477032 7462 consensus_queue.cc:260] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.447669 7459 tablet_bootstrap.cc:492] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:00.484963 7483 ts_tablet_manager.cc:1929] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:00.485581 7459 ts_tablet_manager.cc:1403] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.464s user 0.013s sys 0.003s
I20260317 21:19:00.485523 7483 log.cc:1199] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:00.452508 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b6fb7cb082cf47ce8cf2a98146633af7" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:00.486649 7483 ts_tablet_manager.cc:1950] T 478f55d4c3034be0b1ef336a173c55b2 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:00.489789 7462 ts_tablet_manager.cc:1434] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.019s user 0.009s sys 0.001s
W20260317 21:19:00.466988 6973 leader_election.cc:343] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 589aadc7897b422e90d3f7caed76ab59
I20260317 21:19:00.491079 6973 leader_election.cc:304] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:00.492098 7528 raft_consensus.cc:2749] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:00.495095 7459 raft_consensus.cc:359] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.495808 7459 raft_consensus.cc:385] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.496059 7459 raft_consensus.cc:740] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.496769 7037 raft_consensus.cc:3060] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:00.498629 7459 consensus_queue.cc:260] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.504580 7459 ts_tablet_manager.cc:1434] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260317 21:19:00.505976 7459 tablet_bootstrap.cc:492] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:00.518989 7462 tablet_bootstrap.cc:492] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:00.523314 7467 ts_tablet_manager.cc:1403] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.187s user 0.012s sys 0.000s
I20260317 21:19:00.527324 7717 leader_election.cc:290] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:00.529336 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 88eae1e70bf94ef3ae0aa64706dcb599 consensus state has no leader
W20260317 21:19:00.530720 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 740ae685629041f39bbbc8ce715d662e consensus state has no leader
I20260317 21:19:00.518416 7161 tablet_service.cc:1511] Processing CreateTablet for tablet f9e98c10a83e429a98a0fb4328e1b0ee (DEFAULT_TABLE table=default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.532722 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e98c10a83e429a98a0fb4328e1b0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.532799 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 478f55d4c3034be0b1ef336a173c55b2 (table default.Table_9f4465b6fdf24369a96e3bf37cfd19d7 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]) successfully deleted
I20260317 21:19:00.536324 6674 catalog_manager.cc:5671] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:00.538089 6674 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:19:00.526252 7467 raft_consensus.cc:359] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.541736 7467 raft_consensus.cc:385] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.542104 7467 raft_consensus.cc:740] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.542858 7467 consensus_queue.cc:260] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.544046 7462 tablet_bootstrap.cc:654] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.557091 7467 ts_tablet_manager.cc:1434] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.033s user 0.007s sys 0.000s
W20260317 21:19:00.558215 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 88eae1e70bf94ef3ae0aa64706dcb599 (table default.Table_d34a86e0f32b40099d38cb403d3b5fc8 [id=32b8542f07164e869bd6a8e7ab6767cd], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 88eae1e70bf94ef3ae0aa64706dcb599 consensus state has no leader
I20260317 21:19:00.567351 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 440cb8b112044a53804357056f1226f1 (DEFAULT_TABLE table=default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.567613 7304 tablet_service.cc:1511] Processing CreateTablet for tablet f9e98c10a83e429a98a0fb4328e1b0ee (DEFAULT_TABLE table=default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.568605 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e98c10a83e429a98a0fb4328e1b0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.570724 7467 tablet_bootstrap.cc:492] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:00.568555 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 440cb8b112044a53804357056f1226f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.576978 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 440cb8b112044a53804357056f1226f1 (DEFAULT_TABLE table=default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.578204 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 440cb8b112044a53804357056f1226f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.581983 7462 tablet_bootstrap.cc:492] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:00.586566 7462 ts_tablet_manager.cc:1403] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.004s
I20260317 21:19:00.499501 7791 log.cc:1199] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/c1e672ca556e400ab05763abcd49ad77
I20260317 21:19:00.590852 7791 ts_tablet_manager.cc:1950] T c1e672ca556e400ab05763abcd49ad77 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:00.592124 7459 tablet_bootstrap.cc:654] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.581684 7478 ts_tablet_manager.cc:1929] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:00.593990 7478 log.cc:1199] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/60f3d32e1f7649fa92861f51134cf000
I20260317 21:19:00.594506 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 130) took 1111 ms. Trace:
I20260317 21:19:00.594702 7478 ts_tablet_manager.cc:1950] T 60f3d32e1f7649fa92861f51134cf000 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:00.594660 7015 tablet_service.cc:1511] Processing CreateTablet for tablet f9e98c10a83e429a98a0fb4328e1b0ee (DEFAULT_TABLE table=default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.594980 7791 rpcz_store.cc:276] 0317 21:18:59.483436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.483582 (+ 146us) service_pool.cc:225] Handling call
0317 21:19:00.191562 (+707980us) ts_tablet_manager.cc:1151] Deleting tablet c1e672ca556e400ab05763abcd49ad77
0317 21:19:00.191614 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:00.363215 (+171601us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:00.445537 (+ 82322us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:00.594420 (+148883us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":57849,"tablet-delete.queue_time_us":620281}
I20260317 21:19:00.595885 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f9e98c10a83e429a98a0fb4328e1b0ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.596740 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet c1e672ca556e400ab05763abcd49ad77 (table default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]) successfully deleted
I20260317 21:19:00.597059 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 60f3d32e1f7649fa92861f51134cf000 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]) successfully deleted
I20260317 21:19:00.597389 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 440cb8b112044a53804357056f1226f1 (DEFAULT_TABLE table=default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.597532 7791 tablet_replica.cc:333] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:00.597730 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 06877bd7155842b9a5ae4a23ea68ada0 because the tablet was not found. No further retry: Not found: Tablet not found: 06877bd7155842b9a5ae4a23ea68ada0
W20260317 21:19:00.598310 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 589aadc7897b422e90d3f7caed76ab59 because the tablet was not found. No further retry: Not found: Tablet not found: 589aadc7897b422e90d3f7caed76ab59
I20260317 21:19:00.598460 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 440cb8b112044a53804357056f1226f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.598412 7791 raft_consensus.cc:2243] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
W20260317 21:19:00.598894 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 59fce28afbeb4d8eadf63f9a97d1183d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fce28afbeb4d8eadf63f9a97d1183d already in progress: creating tablet
I20260317 21:19:00.599236 7791 pending_rounds.cc:70] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:00.599491 7478 tablet_replica.cc:333] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:00.599473 7791 pending_rounds.cc:77] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412456470798336 op_type: NO_OP noop_request { }
I20260317 21:19:00.599867 7791 raft_consensus.cc:2889] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:00.600250 7478 raft_consensus.cc:2243] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:00.600281 7791 raft_consensus.cc:2272] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.600801 7478 raft_consensus.cc:2272] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.600982 7467 tablet_bootstrap.cc:654] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.603142 7478 ts_tablet_manager.cc:1916] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.608389 7037 raft_consensus.cc:2468] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:00.619577 7528 consensus_queue.cc:1048] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:00.620271 7120 leader_election.cc:304] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:00.621258 7780 raft_consensus.cc:2804] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:00.621835 7780 raft_consensus.cc:697] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:00.622738 7780 consensus_queue.cc:237] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.633323 7459 tablet_bootstrap.cc:492] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:00.633783 7459 ts_tablet_manager.cc:1403] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.128s user 0.011s sys 0.002s
I20260317 21:19:00.633915 7827 consensus_queue.cc:1048] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:00.634680 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d44ce7d02ab3404389f88fd94db6a9e9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:00.636286 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d44ce7d02ab3404389f88fd94db6a9e9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:00.637694 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a3070bf38c304436b100f2ddb8a4716f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:00.639379 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3070bf38c304436b100f2ddb8a4716f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:00.639214 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b9162607fa9d414e8d894b6d026d45de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:00.641119 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9162607fa9d414e8d894b6d026d45de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:00.640810 7827 raft_consensus.cc:493] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.641546 7827 raft_consensus.cc:515] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.649134 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:00.652351 6673 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 0eee8b5588c24661889c593797c94269
W20260317 21:19:00.656508 6666 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 34) took 752 ms (client timeout 1000 ms). Trace:
W20260317 21:19:00.656926 6666 rpcz_store.cc:269] 0317 21:18:59.904342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.904488 (+ 146us) service_pool.cc:225] Handling call
0317 21:19:00.656430 (+751942us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260317 21:19:00.659950 7014 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:00.663648 7163 tablet_service.cc:1511] Processing CreateTablet for tablet 99c6483618044272890b59f6b7447081 (DEFAULT_TABLE table=default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.664952 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99c6483618044272890b59f6b7447081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.665544 7478 ts_tablet_manager.cc:1929] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:00.672574 7827 leader_election.cc:290] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:00.671829 7478 log.cc:1199] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/c6d5385171364a078d968bb0a489383a
I20260317 21:19:00.667261 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.675686 7478 ts_tablet_manager.cc:1950] T c6d5385171364a078d968bb0a489383a P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:00.678534 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 99c6483618044272890b59f6b7447081 (DEFAULT_TABLE table=default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.679562 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99c6483618044272890b59f6b7447081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.681974 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet c6d5385171364a078d968bb0a489383a (table default.Table_4493a733fd0a496885b7745fcfc6cf2e [id=e8d1164df252476a82baf64ec8c6c9cf]) successfully deleted
I20260317 21:19:00.685052 7478 tablet_replica.cc:333] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:00.685781 7478 raft_consensus.cc:2243] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:00.686610 7478 raft_consensus.cc:2272] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.695199 7478 ts_tablet_manager.cc:1916] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.696040 7467 tablet_bootstrap.cc:492] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:00.696465 7467 ts_tablet_manager.cc:1403] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.126s user 0.008s sys 0.004s
I20260317 21:19:00.635680 7815 raft_consensus.cc:493] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.702207 7815 raft_consensus.cc:515] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.659929 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:00.705561 7483 tablet_replica.cc:333] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:00.706352 7483 raft_consensus.cc:2243] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:00.707134 7483 raft_consensus.cc:2272] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.672477 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f7ec20308a9244a49992e378fa7b4f6c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:00.710856 7326 raft_consensus.cc:2393] T f7ec20308a9244a49992e378fa7b4f6c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:19:00.711231 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 99c6483618044272890b59f6b7447081 (DEFAULT_TABLE table=default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.712993 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99c6483618044272890b59f6b7447081. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.713630 7815 leader_election.cc:290] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.689936 7785 raft_consensus.cc:493] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.716622 7785 raft_consensus.cc:515] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.718968 7785 leader_election.cc:290] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.720978 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:00.722584 7264 leader_election.cc:343] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.724776 7791 ts_tablet_manager.cc:1916] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.725299 7483 ts_tablet_manager.cc:1916] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.736743 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_cf3bed4cf36c4e05a6ca6ab34245a067" } new_table_name: "default.Table_98c6c8ccb3864ea59aefae46b94eb7a1" modify_external_catalogs: true
I20260317 21:19:00.742161 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_a7bea1acda68407ea38b257a2659e413" } new_table_name: "default.Table_ba79dc7832c24e2c980af6335f16d998" modify_external_catalogs: true
I20260317 21:19:00.748323 7478 ts_tablet_manager.cc:1929] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:00.754479 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:00.756138 7478 log.cc:1199] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:00.756721 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:00.753825 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:00.757875 7184 raft_consensus.cc:2468] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:00.758667 7478 ts_tablet_manager.cc:1950] T 478f55d4c3034be0b1ef336a173c55b2 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
W20260317 21:19:00.759982 7261 leader_election.cc:343] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.760531 7261 leader_election.cc:304] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:00.761560 7815 raft_consensus.cc:2749] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:00.764196 7261 leader_election.cc:304] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:00.766323 6671 master_service.cc:954] ReplaceTablet: received request to replace tablet 0eee8b5588c24661889c593797c94269 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:00.774652 7163 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.774892 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 478f55d4c3034be0b1ef336a173c55b2 (table default.Table_9f4465b6fdf24369a96e3bf37cfd19d7 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]) successfully deleted
W20260317 21:19:00.775620 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 0eee8b5588c24661889c593797c94269 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0eee8b5588c24661889c593797c94269 already in progress: creating tablet
I20260317 21:19:00.780118 6672 catalog_manager.cc:5671] T 6f980bd133da4b04b101e8df99ccca6f P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:00.799470 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 35023195b9094d58827921a1dcb2ff05
I20260317 21:19:00.800993 6667 catalog_manager.cc:6741] ReplaceTablet: tablet 35023195b9094d58827921a1dcb2ff05 deleted and replaced by tablet bc2cd63423f844aea29ed3fd839f083e
I20260317 21:19:00.804713 7815 raft_consensus.cc:2804] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:00.805251 7815 raft_consensus.cc:493] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:00.805698 7815 raft_consensus.cc:3060] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:00.806288 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 8d94620882364ed3929ee314814910c9 (DEFAULT_TABLE table=default.Table_0f98ce20efc14dfea9ce425147e744e9 [id=872cbb89b0d1405e9858d37dc22c05c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.807794 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d94620882364ed3929ee314814910c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.608599 7462 raft_consensus.cc:359] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.776957 7307 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:00.817003 7462 raft_consensus.cc:385] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.778345 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 8d94620882364ed3929ee314814910c9 (DEFAULT_TABLE table=default.Table_0f98ce20efc14dfea9ce425147e744e9 [id=872cbb89b0d1405e9858d37dc22c05c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.817565 7462 raft_consensus.cc:740] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.817576 7163 tablet_service.cc:1558] Processing DeleteTablet for tablet 478f55d4c3034be0b1ef336a173c55b2 with delete_type TABLET_DATA_DELETED (replaced by tablet 8a44a297079347bab6f8be6abe332bfa) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.817822 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:00.818761 7478 tablet_replica.cc:333] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:00.818413 7462 consensus_queue.cc:260] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:00.819042 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 3e2f891e712341868d94bab1dd72a9e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e2f891e712341868d94bab1dd72a9e1 already in progress: creating tablet
I20260317 21:19:00.819269 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:00.819718 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 478f55d4c3034be0b1ef336a173c55b2 because the tablet was not found. No further retry: Not found: Tablet not found: 478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:00.821879 7462 ts_tablet_manager.cc:1434] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.215s user 0.003s sys 0.003s
I20260317 21:19:00.825724 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_edca133386d246bda40142e7ce350141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:00.826702 7815 raft_consensus.cc:515] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:00.827903 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_edca133386d246bda40142e7ce350141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:00.829197 7467 raft_consensus.cc:359] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.830276 7467 raft_consensus.cc:385] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:00.830874 7467 raft_consensus.cc:740] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.832072 7467 consensus_queue.cc:260] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.833070 7478 raft_consensus.cc:2243] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:00.839075 7478 pending_rounds.cc:70] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:19:00.839335 7478 pending_rounds.cc:77] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412467438841856 op_type: NO_OP noop_request { }
I20260317 21:19:00.839692 7478 raft_consensus.cc:2889] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:00.750388 6973 leader_election.cc:304] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:00.840041 7478 raft_consensus.cc:2272] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:00.840768 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 90520.7us
I20260317 21:19:00.757128 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:00.762326 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 8d94620882364ed3929ee314814910c9 (DEFAULT_TABLE table=default.Table_0f98ce20efc14dfea9ce425147e744e9 [id=872cbb89b0d1405e9858d37dc22c05c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.842434 7478 ts_tablet_manager.cc:1916] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.843263 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d94620882364ed3929ee314814910c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.850945 7467 ts_tablet_manager.cc:1434] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.154s user 0.007s sys 0.001s
I20260317 21:19:00.854094 7827 raft_consensus.cc:3060] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:00.766469 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:00.802471 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:00.815510 6971 consensus_peers.cc:597] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60f3d32e1f7649fa92861f51134cf000. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:00.863672 7815 leader_election.cc:290] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.833324 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:00.837530 7307 tablet_service.cc:1511] Processing CreateTablet for tablet 740ae685629041f39bbbc8ce715d662e (DEFAULT_TABLE table=default.Table_40785d5a99c543259bad7c3c6bcbce39 [id=f50010703e63443f8ae1ee00b1161dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.822270 7831 raft_consensus.cc:493] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:19:00.827885 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 64489.6us
I20260317 21:19:00.864905 7831 raft_consensus.cc:515] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.867409 7831 leader_election.cc:290] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:00.871307 7827 raft_consensus.cc:2749] T f7ec20308a9244a49992e378fa7b4f6c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:00.870076 6971 leader_election.cc:343] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:19:00.866355 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:00.880890 7184 raft_consensus.cc:3060] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:00.881443 7467 tablet_bootstrap.cc:492] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:00.880154 7478 ts_tablet_manager.cc:1929] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:00.885213 7478 log.cc:1199] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:19:00.886018 7478 ts_tablet_manager.cc:1950] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:00.893059 7184 raft_consensus.cc:2468] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:00.896214 7261 leader_election.cc:304] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:00.897423 7815 raft_consensus.cc:2804] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:00.898222 7815 raft_consensus.cc:697] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:00.899163 7785 raft_consensus.cc:493] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.899762 7785 raft_consensus.cc:515] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.902238 7785 leader_election.cc:290] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.902829 7785 raft_consensus.cc:493] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:00.905586 7785 raft_consensus.cc:515] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.903833 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:00.911885 7307 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740ae685629041f39bbbc8ce715d662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.912213 7483 ts_tablet_manager.cc:1929] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:00.912695 7483 log.cc:1199] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/0eee8b5588c24661889c593797c94269
I20260317 21:19:00.913117 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d94620882364ed3929ee314814910c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.913484 7462 tablet_bootstrap.cc:492] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:00.913683 7483 ts_tablet_manager.cc:1950] T 0eee8b5588c24661889c593797c94269 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:00.916589 7483 tablet_replica.cc:333] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:00.917915 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35023195b9094d58827921a1dcb2ff05" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:00.919101 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:00.919364 7261 leader_election.cc:343] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:00.920984 6973 leader_election.cc:343] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:19:00.921373 6973 leader_election.cc:304] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:00.921953 7261 leader_election.cc:343] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:00.922129 7827 raft_consensus.cc:2749] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:00.917394 7483 raft_consensus.cc:2243] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
W20260317 21:19:00.922470 6973 consensus_peers.cc:597] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 60f3d32e1f7649fa92861f51134cf000. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:00.911330 7785 leader_election.cc:290] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:00.923007 7483 raft_consensus.cc:2272] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:00.926291 7483 ts_tablet_manager.cc:1916] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:00.932591 7467 tablet_bootstrap.cc:654] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:00.935611 6662 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 81621us
W20260317 21:19:00.939958 7829 log.cc:927] Time spent T 6f980bd133da4b04b101e8df99ccca6f P c0ef92fef382401a9564c17e0c223f6a: Append to log took a long time: real 0.207s user 0.001s sys 0.000s
I20260317 21:19:00.943737 7462 tablet_bootstrap.cc:654] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:00.940363 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 740ae685629041f39bbbc8ce715d662e (DEFAULT_TABLE table=default.Table_40785d5a99c543259bad7c3c6bcbce39 [id=f50010703e63443f8ae1ee00b1161dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.953685 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 18e4b69574e44eb28a735edc64e97542 (DEFAULT_TABLE table=default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.954237 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740ae685629041f39bbbc8ce715d662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.954886 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e4b69574e44eb28a735edc64e97542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.952062 7163 tablet_service.cc:1511] Processing CreateTablet for tablet b2dccb915fad4666bd7978f8c416db44 (DEFAULT_TABLE table=default.Table_94ac54eae69d40e6974b829a10d006ab [id=15b94f5175ad4ddbad9b6fe758ae3ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.956393 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dccb915fad4666bd7978f8c416db44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.937620 7815 consensus_queue.cc:237] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.973400 6668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_c82fde1f1c4e4ff4ab059d998ab8399a" } modify_external_catalogs: true
I20260317 21:19:00.974045 6668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_c82fde1f1c4e4ff4ab059d998ab8399a" } modify_external_catalogs: true
I20260317 21:19:00.977756 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1c93ac734b4d4ff68afbd9c72d470944" } modify_external_catalogs: true
I20260317 21:19:00.978533 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_1c93ac734b4d4ff68afbd9c72d470944" } modify_external_catalogs: true
I20260317 21:19:00.947140 7483 ts_tablet_manager.cc:1929] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:00.980029 7483 log.cc:1199] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/35023195b9094d58827921a1dcb2ff05
I20260317 21:19:00.982996 7483 ts_tablet_manager.cc:1950] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:00.986925 7011 tablet_service.cc:1511] Processing CreateTablet for tablet 88eae1e70bf94ef3ae0aa64706dcb599 (DEFAULT_TABLE table=default.Table_d34a86e0f32b40099d38cb403d3b5fc8 [id=32b8542f07164e869bd6a8e7ab6767cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.990141 7018 tablet_service.cc:1511] Processing CreateTablet for tablet 722d940aee014df4aeaddbc7aace4f4e (DEFAULT_TABLE table=default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.992337 7010 tablet_service.cc:1511] Processing CreateTablet for tablet a4cee7952cf5428b98340789a48d2da2 (DEFAULT_TABLE table=default.Table_a571e2a5fe2e4b1c92f568eb955c8a38 [id=2652ee9fdb984f14a138c570bda4636d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.994613 7013 tablet_service.cc:1511] Processing CreateTablet for tablet e586f11c37184506a15d726b4eb333be (DEFAULT_TABLE table=default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.636163 7459 raft_consensus.cc:359] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:00.997846 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 18e4b69574e44eb28a735edc64e97542 (DEFAULT_TABLE table=default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.999671 7016 tablet_service.cc:1511] Processing CreateTablet for tablet b2dccb915fad4666bd7978f8c416db44 (DEFAULT_TABLE table=default.Table_94ac54eae69d40e6974b829a10d006ab [id=15b94f5175ad4ddbad9b6fe758ae3ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.002211 7006 tablet_service.cc:1511] Processing CreateTablet for tablet e06a1488c60f4d1387417ce30579fbf4 (DEFAULT_TABLE table=default.Table_c1eb7feff9ea4b568cb56bb3761a2349 [id=3e906ea0e8af47e383d0403c155be644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.996351 7017 tablet_service.cc:1511] Processing CreateTablet for tablet 252f728d183f440a9ef42894508d5755 (DEFAULT_TABLE table=default.Table_c846e042b2d4448ab3aa5e50bda65ec4 [id=2c877c93e6cc4a76baca68194014e884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.006593 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 2a221dfad830428db2329d6e543f41dc (DEFAULT_TABLE table=default.Table_409b88634b48453daacc460109b51caf [id=2d22abe029d348c484e6aff5df8b3d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:00.998082 7459 raft_consensus.cc:385] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:01.007884 7459 raft_consensus.cc:740] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:00.988489 7011 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88eae1e70bf94ef3ae0aa64706dcb599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.012966 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e586f11c37184506a15d726b4eb333be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.018227 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a221dfad830428db2329d6e543f41dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.019935 7843 tablet.cc:1723] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:01.023200 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 18e4b69574e44eb28a735edc64e97542 (DEFAULT_TABLE table=default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.024335 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e4b69574e44eb28a735edc64e97542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.024811 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 18e4b69574e44eb28a735edc64e97542. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.029198 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252f728d183f440a9ef42894508d5755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.031140 7462 tablet_bootstrap.cc:492] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:01.031584 7462 ts_tablet_manager.cc:1403] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.118s user 0.012s sys 0.011s
I20260317 21:19:01.033102 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4cee7952cf5428b98340789a48d2da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.034121 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 88eae1e70bf94ef3ae0aa64706dcb599 (DEFAULT_TABLE table=default.Table_d34a86e0f32b40099d38cb403d3b5fc8 [id=32b8542f07164e869bd6a8e7ab6767cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.035297 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88eae1e70bf94ef3ae0aa64706dcb599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.037024 7459 consensus_queue.cc:260] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.037370 7305 tablet_service.cc:1511] Processing CreateTablet for tablet b2dccb915fad4666bd7978f8c416db44 (DEFAULT_TABLE table=default.Table_94ac54eae69d40e6974b829a10d006ab [id=15b94f5175ad4ddbad9b6fe758ae3ab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.038416 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dccb915fad4666bd7978f8c416db44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:00.837334 7014 tablet_service.cc:1511] Processing CreateTablet for tablet 740ae685629041f39bbbc8ce715d662e (DEFAULT_TABLE table=default.Table_40785d5a99c543259bad7c3c6bcbce39 [id=f50010703e63443f8ae1ee00b1161dd2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.043236 7014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 740ae685629041f39bbbc8ce715d662e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.048995 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dccb915fad4666bd7978f8c416db44. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.053345 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06a1488c60f4d1387417ce30579fbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.054767 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 722d940aee014df4aeaddbc7aace4f4e (DEFAULT_TABLE table=default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.056548 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 722d940aee014df4aeaddbc7aace4f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.057104 7018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 722d940aee014df4aeaddbc7aace4f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.061971 7459 ts_tablet_manager.cc:1434] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.428s user 0.006s sys 0.000s
I20260317 21:19:01.064966 7462 raft_consensus.cc:359] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:00.984628 6662 consensus_peers.cc:597] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 -> Peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Couldn't send request to peer 47e1c983d0224428a9a7b147a0e5e4ef. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111). This is attempt 21: this message will repeat every 5th retry.
I20260317 21:19:01.010944 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e48e213b99454430b7ede635351ef076"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:01.070550 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e48e213b99454430b7ede635351ef076 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:00.997727 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet b6fb7cb082cf47ce8cf2a98146633af7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]) successfully deleted
W20260317 21:19:01.076016 6659 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 78308.2us
W20260317 21:19:01.076761 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 35023195b9094d58827921a1dcb2ff05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35023195b9094d58827921a1dcb2ff05 already in progress: creating tablet
W20260317 21:19:01.076993 7443 thread.cc:641] rpc reactor (reactor) Time spent creating pthread: real 0.540s user 0.000s sys 0.013s
W20260317 21:19:01.077392 7443 thread.cc:608] rpc reactor (reactor) Time spent starting thread: real 0.540s user 0.000s sys 0.013s
I20260317 21:19:01.078872 7302 tablet_service.cc:1511] Processing CreateTablet for tablet a4cee7952cf5428b98340789a48d2da2 (DEFAULT_TABLE table=default.Table_a571e2a5fe2e4b1c92f568eb955c8a38 [id=2652ee9fdb984f14a138c570bda4636d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.081902 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4cee7952cf5428b98340789a48d2da2. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:01.077257 6662 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 92715.3us
I20260317 21:19:01.085248 7831 raft_consensus.cc:493] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:01.102301 7300 tablet_service.cc:1511] Processing CreateTablet for tablet e586f11c37184506a15d726b4eb333be (DEFAULT_TABLE table=default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.103495 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e586f11c37184506a15d726b4eb333be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.110071 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 88eae1e70bf94ef3ae0aa64706dcb599 (DEFAULT_TABLE table=default.Table_d34a86e0f32b40099d38cb403d3b5fc8 [id=32b8542f07164e869bd6a8e7ab6767cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.111939 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 88eae1e70bf94ef3ae0aa64706dcb599. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.113286 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 722d940aee014df4aeaddbc7aace4f4e (DEFAULT_TABLE table=default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.114522 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 722d940aee014df4aeaddbc7aace4f4e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:01.114827 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 564d41fc13b04ec688fc38a78f55e83d consensus state has no leader
I20260317 21:19:01.115834 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bc634ee1d0df4afcb027ea787dce81b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:01.117470 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc634ee1d0df4afcb027ea787dce81b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:01.122040 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7d73f093d0e8434ca4c41cef6dfe310c consensus state has no leader
I20260317 21:19:01.123891 7462 raft_consensus.cc:385] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:01.126071 7306 tablet_service.cc:1511] Processing CreateTablet for tablet 252f728d183f440a9ef42894508d5755 (DEFAULT_TABLE table=default.Table_c846e042b2d4448ab3aa5e50bda65ec4 [id=2c877c93e6cc4a76baca68194014e884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.129526 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252f728d183f440a9ef42894508d5755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.140799 7462 raft_consensus.cc:740] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:01.144322 7467 tablet_bootstrap.cc:492] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:01.144821 7467 ts_tablet_manager.cc:1403] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.264s user 0.004s sys 0.008s
I20260317 21:19:01.144335 7308 tablet_service.cc:1511] Processing CreateTablet for tablet e06a1488c60f4d1387417ce30579fbf4 (DEFAULT_TABLE table=default.Table_c1eb7feff9ea4b568cb56bb3761a2349 [id=3e906ea0e8af47e383d0403c155be644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.146370 7308 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06a1488c60f4d1387417ce30579fbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.157101 7156 tablet_service.cc:1511] Processing CreateTablet for tablet e586f11c37184506a15d726b4eb333be (DEFAULT_TABLE table=default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.158512 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e586f11c37184506a15d726b4eb333be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.162444 7160 tablet_service.cc:1511] Processing CreateTablet for tablet a4cee7952cf5428b98340789a48d2da2 (DEFAULT_TABLE table=default.Table_a571e2a5fe2e4b1c92f568eb955c8a38 [id=2652ee9fdb984f14a138c570bda4636d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.163843 7160 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4cee7952cf5428b98340789a48d2da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.164499 7462 consensus_queue.cc:260] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.169967 7157 tablet_service.cc:1511] Processing CreateTablet for tablet 2a221dfad830428db2329d6e543f41dc (DEFAULT_TABLE table=default.Table_409b88634b48453daacc460109b51caf [id=2d22abe029d348c484e6aff5df8b3d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.170782 7164 tablet_service.cc:1511] Processing CreateTablet for tablet e06a1488c60f4d1387417ce30579fbf4 (DEFAULT_TABLE table=default.Table_c1eb7feff9ea4b568cb56bb3761a2349 [id=3e906ea0e8af47e383d0403c155be644]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.171407 7157 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a221dfad830428db2329d6e543f41dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.172127 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e06a1488c60f4d1387417ce30579fbf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.175305 7153 tablet_service.cc:1511] Processing CreateTablet for tablet 203f33157a0d4b82aab44da452aa46dd (DEFAULT_TABLE table=default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.177069 7153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203f33157a0d4b82aab44da452aa46dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.178300 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 203f33157a0d4b82aab44da452aa46dd (DEFAULT_TABLE table=default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.179320 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203f33157a0d4b82aab44da452aa46dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.180948 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 252f728d183f440a9ef42894508d5755 (DEFAULT_TABLE table=default.Table_c846e042b2d4448ab3aa5e50bda65ec4 [id=2c877c93e6cc4a76baca68194014e884]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.181964 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252f728d183f440a9ef42894508d5755. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.178241 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 203f33157a0d4b82aab44da452aa46dd (DEFAULT_TABLE table=default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.188539 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 203f33157a0d4b82aab44da452aa46dd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.141162 7443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:19:01.186345 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 8744f6237e154dd99bc4c7e54822f7be (DEFAULT_TABLE table=default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.193763 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8744f6237e154dd99bc4c7e54822f7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.195741 7443 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:19:01.196341 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:19:01.158128 7831 raft_consensus.cc:515] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.212467 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:01.213140 7329 raft_consensus.cc:2468] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:01.214402 6973 leader_election.cc:304] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:01.215209 7827 raft_consensus.cc:2804] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:01.215416 7831 leader_election.cc:290] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:01.215550 7827 raft_consensus.cc:493] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:01.215883 7827 raft_consensus.cc:3060] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:01.187216 7264 leader_election.cc:343] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:01.225824 7264 leader_election.cc:343] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.238565 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 2a221dfad830428db2329d6e543f41dc (DEFAULT_TABLE table=default.Table_409b88634b48453daacc460109b51caf [id=2d22abe029d348c484e6aff5df8b3d29]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.239692 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2a221dfad830428db2329d6e543f41dc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.250370 7459 tablet_bootstrap.cc:492] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:01.251165 7462 ts_tablet_manager.cc:1434] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.219s user 0.004s sys 0.008s
I20260317 21:19:01.253088 7780 raft_consensus.cc:493] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:01.253788 7780 raft_consensus.cc:515] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.260530 7462 tablet_bootstrap.cc:492] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:01.265488 7162 consensus_meta.cc:287] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: Time spent flushing consensus metadata: real 0.602s user 0.009s sys 0.000s
W20260317 21:19:01.269622 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 87069.5us
I20260317 21:19:01.275127 7827 raft_consensus.cc:515] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.277590 7827 leader_election.cc:290] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:01.255944 7780 leader_election.cc:290] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:01.287789 7467 raft_consensus.cc:359] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.297405 7467 raft_consensus.cc:385] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:01.297691 7467 raft_consensus.cc:740] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:01.297595 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35023195b9094d58827921a1dcb2ff05" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:01.298411 7467 consensus_queue.cc:260] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.300127 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:01.300495 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:01.300809 7327 raft_consensus.cc:3060] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:01.302156 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:01.302883 7035 raft_consensus.cc:2393] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:01.326006 7842 tablet.cc:1723] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260317 21:19:01.330613 7264 leader_election.cc:343] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.331048 7264 leader_election.cc:304] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:01.331565 7264 raft_consensus.cc:2686] T 35023195b9094d58827921a1dcb2ff05 P 2f539d1a4a584b8ca83928be49369bf4: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:19:01.293071 7820 consensus_queue.cc:1048] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
W20260317 21:19:01.365623 7119 leader_election.cc:343] T b6fb7cb082cf47ce8cf2a98146633af7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:19:01.380093 7467 ts_tablet_manager.cc:1434] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.235s user 0.005s sys 0.002s
I20260317 21:19:01.381353 7467 tablet_bootstrap.cc:492] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:01.412829 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 0eee8b5588c24661889c593797c94269 (table default.Table_212ed8405de344519eaecc32555fcbb5 [id=8fee88b3acf649ffbfbc18d16f47af40]) successfully deleted
W20260317 21:19:01.416917 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 86331.5us
W20260317 21:19:01.417645 7264 leader_election.cc:343] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.418028 7264 leader_election.cc:304] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:01.422684 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 7d73f093d0e8434ca4c41cef6dfe310c (DEFAULT_TABLE table=default.Table_ba79dc7832c24e2c980af6335f16d998 [id=d81f432ebf45465c9543274e89aaadd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.424476 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d73f093d0e8434ca4c41cef6dfe310c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:01.424619 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 3e2f891e712341868d94bab1dd72a9e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e2f891e712341868d94bab1dd72a9e1 already in progress: creating tablet
I20260317 21:19:01.427012 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 35023195b9094d58827921a1dcb2ff05 (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]) successfully deleted
I20260317 21:19:01.428016 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 7d73f093d0e8434ca4c41cef6dfe310c (DEFAULT_TABLE table=default.Table_ba79dc7832c24e2c980af6335f16d998 [id=d81f432ebf45465c9543274e89aaadd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.429037 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d73f093d0e8434ca4c41cef6dfe310c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.429692 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 8744f6237e154dd99bc4c7e54822f7be (DEFAULT_TABLE table=default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.431334 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8744f6237e154dd99bc4c7e54822f7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.431394 7155 tablet_service.cc:1511] Processing CreateTablet for tablet efd2f45acf114e3ca577103de845c5a1 (DEFAULT_TABLE table=default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.434546 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e (DEFAULT_TABLE table=default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.435192 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 8744f6237e154dd99bc4c7e54822f7be (DEFAULT_TABLE table=default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.436307 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8744f6237e154dd99bc4c7e54822f7be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.438287 7300 tablet_service.cc:1511] Processing CreateTablet for tablet efd2f45acf114e3ca577103de845c5a1 (DEFAULT_TABLE table=default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.439446 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd2f45acf114e3ca577103de845c5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.440042 7462 tablet_bootstrap.cc:654] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:01.443365 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 3d15d835328d4f7b8fea41cbedf203fe (DEFAULT_TABLE table=default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.445138 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d15d835328d4f7b8fea41cbedf203fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.445075 7844 raft_consensus.cc:2749] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:01.446161 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e (DEFAULT_TABLE table=default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.447211 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.426075 7820 consensus_queue.cc:1048] T f7ec20308a9244a49992e378fa7b4f6c P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260317 21:19:01.449266 7007 tablet_service.cc:1511] Processing CreateTablet for tablet e22772281bd74754bb969e87a70d8077 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.450330 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22772281bd74754bb969e87a70d8077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.450341 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 3d15d835328d4f7b8fea41cbedf203fe (DEFAULT_TABLE table=default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.450717 7152 tablet_service.cc:1511] Processing CreateTablet for tablet 7fc38947219047b5abdf71af5554a987 (DEFAULT_TABLE table=default.Table_ef1aaeadf5354f7e99d9db422dfc6e63 [id=46ad456ffb374be585c1a222df6dabc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.460700 7163 tablet_service.cc:1511] Processing CreateTablet for tablet f3b5825c77ac4d69a39978932051db8c (DEFAULT_TABLE table=default.Table_f2bf8c9535e94059a16a46b599e39416 [id=043de1466ade4509ab0fb8a99eba703a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.472751 7306 tablet_service.cc:1511] Processing CreateTablet for tablet f3b5825c77ac4d69a39978932051db8c (DEFAULT_TABLE table=default.Table_f2bf8c9535e94059a16a46b599e39416 [id=043de1466ade4509ab0fb8a99eba703a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.474098 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3b5825c77ac4d69a39978932051db8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.474550 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 3d15d835328d4f7b8fea41cbedf203fe (DEFAULT_TABLE table=default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.475576 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d15d835328d4f7b8fea41cbedf203fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.477751 7151 tablet_service.cc:1511] Processing CreateTablet for tablet e22772281bd74754bb969e87a70d8077 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.483220 7017 tablet_service.cc:1511] Processing CreateTablet for tablet a4d64c48641b484ab443380808544d2f (DEFAULT_TABLE table=default.Table_b8f93eb9b7b047cf9b6604988438ec76 [id=f966bc12b8824b708f7ff3d9f5789eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.484297 7017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d64c48641b484ab443380808544d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.486997 7009 tablet_service.cc:1511] Processing CreateTablet for tablet f3b5825c77ac4d69a39978932051db8c (DEFAULT_TABLE table=default.Table_f2bf8c9535e94059a16a46b599e39416 [id=043de1466ade4509ab0fb8a99eba703a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.488018 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3b5825c77ac4d69a39978932051db8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.488858 7459 tablet_bootstrap.cc:654] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:01.493278 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd2f45acf114e3ca577103de845c5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.493381 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 (DEFAULT_TABLE table=default.Table_0a9cb20926ea45aaa419518e3f9c57ab [id=c5ec46478903444e8777ecd0a8c4b234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.494153 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.494796 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f34026fd2f4575bb86a9dadb733fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.484699 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 7fc38947219047b5abdf71af5554a987 (DEFAULT_TABLE table=default.Table_ef1aaeadf5354f7e99d9db422dfc6e63 [id=46ad456ffb374be585c1a222df6dabc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.505788 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 (DEFAULT_TABLE table=default.Table_0a9cb20926ea45aaa419518e3f9c57ab [id=c5ec46478903444e8777ecd0a8c4b234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.508287 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fc38947219047b5abdf71af5554a987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.509536 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fc38947219047b5abdf71af5554a987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.511592 6675 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
W20260317 21:19:01.514786 6672 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 53) took 1776 ms (client timeout 1000 ms). Trace:
I20260317 21:19:01.516350 7467 tablet_bootstrap.cc:654] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:01.518229 7005 tablet_service.cc:1558] Processing DeleteTablet for tablet 06877bd7155842b9a5ae4a23ea68ada0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:58 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:01.519894 6666 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_b451641b2795486b8d68f72ab543b5b2 [id=22768a64f9394820997448f0c81605ee]). Expected version 1 got 0
I20260317 21:19:01.519121 7820 raft_consensus.cc:493] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:01.523564 7820 raft_consensus.cc:515] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.524482 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3b5825c77ac4d69a39978932051db8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.526335 7820 leader_election.cc:290] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:01.529208 7151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22772281bd74754bb969e87a70d8077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.532732 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d15d835328d4f7b8fea41cbedf203fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:01.515200 6672 rpcz_store.cc:269] 0317 21:18:59.738056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.772965 (+ 34909us) service_pool.cc:225] Handling call
0317 21:19:01.514574 (+1741609us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:01.133357 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:01.133781 (+ 424us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:01.133809 (+ 28us) write_op.cc:435] Acquired schema lock
0317 21:19:01.133931 (+ 122us) tablet.cc:661] Decoding operations
0317 21:19:01.134249 (+ 318us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:01.134288 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:01.134310 (+ 22us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:01.134447 (+ 137us) tablet.cc:700] Row locks acquired
0317 21:19:01.134460 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:01.134582 (+ 122us) write_op.cc:270] Start()
0317 21:19:01.134672 (+ 90us) write_op.cc:276] Timestamp: P: 1773782341134512 usec, L: 0
0317 21:19:01.134685 (+ 13us) op_driver.cc:348] REPLICATION: starting
0317 21:19:01.312890 (+178205us) log.cc:844] Serialized 530 byte log entry
0317 21:19:01.502734 (+189844us) op_driver.cc:464] REPLICATION: finished
0317 21:19:01.503683 (+ 949us) write_op.cc:301] APPLY: starting
0317 21:19:01.503887 (+ 204us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:01.504031 (+ 144us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:01.504168 (+ 137us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:01.504864 (+ 696us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:01.505028 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:01.505077 (+ 49us) write_op.cc:312] APPLY: finished
0317 21:19:01.505336 (+ 259us) log.cc:844] Serialized 23 byte log entry
0317 21:19:01.505511 (+ 175us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:01.505626 (+ 115us) write_op.cc:454] Released schema lock
0317 21:19:01.505672 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:01.505987 (+ 315us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:01.506010 (+ 23us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":4,"child_traces":[["op",{"apply.queue_time_us":692,"apply.run_cpu_time_us":2195,"apply.run_wall_time_us":2302,"num_ops":1,"prepare.queue_time_us":350092,"prepare.run_cpu_time_us":2279,"prepare.run_wall_time_us":180717,"raft.queue_time_us":137,"raft.run_cpu_time_us":579,"raft.run_wall_time_us":580,"replication_time_us":367918}]]}
W20260317 21:19:01.535300 6666 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 35) took 864 ms (client timeout 1000 ms). Trace:
W20260317 21:19:01.535719 6666 rpcz_store.cc:269] 0317 21:19:00.671020 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:00.671143 (+ 123us) service_pool.cc:225] Handling call
0317 21:19:01.535226 (+864083us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5}
I20260317 21:19:01.537091 7004 tablet_service.cc:1558] Processing DeleteTablet for tablet b6fb7cb082cf47ce8cf2a98146633af7 with delete_type TABLET_DATA_DELETED (replaced by tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:01.537541 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet e858a01bc4594cc58853ea3239a8dbad from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:01.537937 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82f8dd260a134810a50465e4a77848ca" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:01.539439 7120 leader_election.cc:343] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.543015 7004 tablet_service.cc:1558] Processing DeleteTablet for tablet 589aadc7897b422e90d3f7caed76ab59 with delete_type TABLET_DATA_DELETED (replaced by tablet 8744f6237e154dd99bc4c7e54822f7be) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:01.545702 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:01.545431 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_fa44c16b78054e6fb846b384a1c68805"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:01.546329 7183 raft_consensus.cc:2468] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:01.548061 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:01.548605 7183 raft_consensus.cc:3060] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:01.553479 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fa44c16b78054e6fb846b384a1c68805 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:01.560950 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f34026fd2f4575bb86a9dadb733fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.561501 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0f98ce20efc14dfea9ce425147e744e9" } new_table_name: "default.Table_d99f91a2b9224cb9a922e32c03228878" modify_external_catalogs: true
I20260317 21:19:01.560768 7183 raft_consensus.cc:2468] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:01.565114 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7d73f093d0e8434ca4c41cef6dfe310c (table default.Table_ba79dc7832c24e2c980af6335f16d998 [id=d81f432ebf45465c9543274e89aaadd1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7d73f093d0e8434ca4c41cef6dfe310c consensus state has no leader
I20260317 21:19:01.565316 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 7d73f093d0e8434ca4c41cef6dfe310c (DEFAULT_TABLE table=default.Table_ba79dc7832c24e2c980af6335f16d998 [id=d81f432ebf45465c9543274e89aaadd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.571393 7161 tablet_service.cc:1511] Processing CreateTablet for tablet a4d64c48641b484ab443380808544d2f (DEFAULT_TABLE table=default.Table_b8f93eb9b7b047cf9b6604988438ec76 [id=f966bc12b8824b708f7ff3d9f5789eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.573133 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d64c48641b484ab443380808544d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.586685 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 3d40549347f8463797d6791136883371
I20260317 21:19:01.566754 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d73f093d0e8434ca4c41cef6dfe310c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.591013 7018 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d40549347f8463797d6791136883371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:01.578960 7015 tablet_service.cc:1511] Processing CreateTablet for tablet efd2f45acf114e3ca577103de845c5a1 (DEFAULT_TABLE table=default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.593668 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efd2f45acf114e3ca577103de845c5a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.444290 7327 raft_consensus.cc:2468] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:01.605532 6973 leader_election.cc:304] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:01.607323 7831 raft_consensus.cc:2804] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:01.607748 7831 raft_consensus.cc:697] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:01.608562 7831 consensus_queue.cc:237] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.612883 7297 tablet_service.cc:1511] Processing CreateTablet for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e (DEFAULT_TABLE table=default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.625366 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b451641b2795486b8d68f72ab543b5b2" } new_table_name: "default.Table_f2b28cd651c443b58e353fb0ccfd33ed" modify_external_catalogs: true
I20260317 21:19:01.629492 7297 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.642261 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 7fc38947219047b5abdf71af5554a987 (DEFAULT_TABLE table=default.Table_ef1aaeadf5354f7e99d9db422dfc6e63 [id=46ad456ffb374be585c1a222df6dabc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.643416 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7fc38947219047b5abdf71af5554a987. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.626456 6668 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 339c3c3c6ade41178620b17ccfaf6081
I20260317 21:19:01.647020 7303 tablet_service.cc:1511] Processing CreateTablet for tablet a4d64c48641b484ab443380808544d2f (DEFAULT_TABLE table=default.Table_b8f93eb9b7b047cf9b6604988438ec76 [id=f966bc12b8824b708f7ff3d9f5789eb8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.649701 7302 tablet_service.cc:1511] Processing CreateTablet for tablet e22772281bd74754bb969e87a70d8077 (DEFAULT_TABLE table=default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.655263 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4d64c48641b484ab443380808544d2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.661731 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 (DEFAULT_TABLE table=default.Table_0a9cb20926ea45aaa419518e3f9c57ab [id=c5ec46478903444e8777ecd0a8c4b234]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:01.663264 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f34026fd2f4575bb86a9dadb733fe0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.665405 7844 raft_consensus.cc:493] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:01.665953 7844 raft_consensus.cc:515] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.669834 7844 leader_election.cc:290] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:01.673115 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e22772281bd74754bb969e87a70d8077. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:01.679751 7791 ts_tablet_manager.cc:1929] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:01.680330 7791 log.cc:1199] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/d59deea79cfb4a83831bb0227797c690
I20260317 21:19:01.681953 7791 ts_tablet_manager.cc:1950] T d59deea79cfb4a83831bb0227797c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:01.684689 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 131) took 2199 ms. Trace:
I20260317 21:19:01.688692 7299 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:01.693804 7462 tablet_bootstrap.cc:492] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:01.694269 7462 ts_tablet_manager.cc:1403] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.434s user 0.015s sys 0.000s
I20260317 21:19:01.696935 7462 raft_consensus.cc:359] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.697666 7462 raft_consensus.cc:385] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:01.697979 7462 raft_consensus.cc:740] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:01.698731 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e858a01bc4594cc58853ea3239a8dbad" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:01.699676 7329 raft_consensus.cc:2393] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:01.710419 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d40549347f8463797d6791136883371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:01.712352 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:01.715695 7261 leader_election.cc:343] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.698869 7462 consensus_queue.cc:260] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.756971 7462 ts_tablet_manager.cc:1434] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.062s user 0.006s sys 0.001s
I20260317 21:19:01.757877 7462 tablet_bootstrap.cc:492] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:01.782557 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 564a02888d854ea58d7a590cba816b86 consensus state has no leader
W20260317 21:19:01.789927 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 3d40549347f8463797d6791136883371 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d40549347f8463797d6791136883371 already in progress: creating tablet
I20260317 21:19:01.789863 7844 consensus_queue.cc:1048] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:01.695528 7827 raft_consensus.cc:493] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:01.801016 7827 raft_consensus.cc:515] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.807538 7827 leader_election.cc:290] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:01.813359 7842 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:42830 (request call id 153) took 1229 ms. Trace:
I20260317 21:19:01.746609 7459 tablet_bootstrap.cc:492] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:01.815330 7459 ts_tablet_manager.cc:1403] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.565s user 0.010s sys 0.008s
I20260317 21:19:01.818696 7459 raft_consensus.cc:359] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.819657 7459 raft_consensus.cc:385] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:01.820039 7459 raft_consensus.cc:740] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:01.826321 6973 leader_election.cc:343] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:19:01.827831 7119 leader_election.cc:304] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:01.829196 7780 raft_consensus.cc:2749] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:01.830863 7119 leader_election.cc:343] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 82f8dd260a134810a50465e4a77848ca
I20260317 21:19:01.831425 7119 leader_election.cc:304] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:01.832176 7780 raft_consensus.cc:2749] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:01.838450 7844 raft_consensus.cc:493] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:01.838940 7844 raft_consensus.cc:515] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:01.841538 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:01.842401 7844 leader_election.cc:290] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:01.844328 6971 leader_election.cc:343] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:19:01.844805 6971 leader_election.cc:304] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:01.847824 7827 raft_consensus.cc:2749] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:01.851584 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d40549347f8463797d6791136883371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:01.849817 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78e2e20d320547b3bb1e03923de52822" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:01.864286 7261 leader_election.cc:343] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.813730 7842 rpcz_store.cc:276] 0317 21:19:00.583775 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:00.583911 (+ 136us) service_pool.cc:225] Handling call
0317 21:19:01.813277 (+1229366us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:00.732550 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0317 21:19:00.768681 (+ 36131us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0317 21:19:00.768721 (+ 40us) alter_schema_op.cc:66] Acquired schema lock
0317 21:19:00.768774 (+ 53us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0317 21:19:00.769073 (+ 299us) alter_schema_op.cc:128] START. Timestamp: 1773782340768921
0317 21:19:00.769099 (+ 26us) op_driver.cc:348] REPLICATION: starting
0317 21:19:00.770318 (+ 1219us) log.cc:844] Serialized 173 byte log entry
0317 21:19:00.885694 (+115376us) op_driver.cc:464] REPLICATION: finished
0317 21:19:01.325862 (+440168us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0317 21:19:01.327523 (+ 1661us) log.cc:844] Serialized 5 byte log entry
0317 21:19:01.736562 (+409039us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:01.785679 (+ 49117us) log.cc:844] Serialized 14 byte log entry
0317 21:19:01.786140 (+ 461us) alter_schema_op.cc:73] Released schema lock
0317 21:19:01.786345 (+ 205us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":439961,"prepare.queue_time_us":147680,"prepare.run_cpu_time_us":3759,"prepare.run_wall_time_us":114486,"raft.queue_time_us":1918,"raft.run_cpu_time_us":2199,"raft.run_wall_time_us":2196,"replication_time_us":116497,"spinlock_wait_cycles":730368,"thread_start_us":609492,"threads_started":3,"wal-append.queue_time_us":42868}]]}
I20260317 21:19:01.880908 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:01.901425 7467 tablet_bootstrap.cc:492] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:01.901968 7467 ts_tablet_manager.cc:1403] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.521s user 0.019s sys 0.014s
W20260317 21:19:01.912133 7264 leader_election.cc:343] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:01.913260 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78e2e20d320547b3bb1e03923de52822" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:01.922235 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d34a86e0f32b40099d38cb403d3b5fc8" } modify_external_catalogs: true
W20260317 21:19:01.923810 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d94620882364ed3929ee314814910c9 consensus state has no leader
I20260317 21:19:01.926792 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet e858a01bc4594cc58853ea3239a8dbad
I20260317 21:19:01.820627 7791 rpcz_store.cc:276] 0317 21:18:59.485355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.485665 (+ 310us) service_pool.cc:225] Handling call
0317 21:19:00.597385 (+1111720us) ts_tablet_manager.cc:1151] Deleting tablet d59deea79cfb4a83831bb0227797c690
0317 21:19:00.597427 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:01.295430 (+698003us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:01.679499 (+384069us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:01.684571 (+ 5072us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1025828}
I20260317 21:19:01.928226 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_62af41ad49db4cfc9d735f36dc9c7666"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:01.928886 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_94ac54eae69d40e6974b829a10d006ab" } modify_external_catalogs: true
I20260317 21:19:01.929384 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_94ac54eae69d40e6974b829a10d006ab" } modify_external_catalogs: true
I20260317 21:19:01.929940 7791 tablet_replica.cc:333] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:01.930460 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_62af41ad49db4cfc9d735f36dc9c7666 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:01.930756 7791 raft_consensus.cc:2243] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:01.931252 7791 raft_consensus.cc:2272] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:01.932979 6661 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 53042.2us
W20260317 21:19:01.934832 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 0eee8b5588c24661889c593797c94269 because the tablet was not found. No further retry: Not found: Tablet not found: 0eee8b5588c24661889c593797c94269
I20260317 21:19:01.935235 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d34a86e0f32b40099d38cb403d3b5fc8" } modify_external_catalogs: true
W20260317 21:19:01.935487 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 3d40549347f8463797d6791136883371 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d40549347f8463797d6791136883371 already in progress: creating tablet
I20260317 21:19:01.928248 6666 catalog_manager.cc:6741] ReplaceTablet: tablet e858a01bc4594cc58853ea3239a8dbad deleted and replaced by tablet b9d1076839894509805be59b751aa847
I20260317 21:19:01.968967 7459 consensus_queue.cc:260] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:01.970732 6662 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 111794us
I20260317 21:19:01.949442 7462 tablet_bootstrap.cc:654] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:01.974339 7459 ts_tablet_manager.cc:1434] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.158s user 0.006s sys 0.001s
I20260317 21:19:01.974779 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet d59deea79cfb4a83831bb0227797c690 (table default.Table_cae455ad5e214610bcfbeff2d7054eff [id=32d051aabc9545b38b462d7ba9f5fd05]) successfully deleted
I20260317 21:19:01.974933 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet e858a01bc4594cc58853ea3239a8dbad with delete_type TABLET_DATA_DELETED (replaced by tablet b9d1076839894509805be59b751aa847) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:01.980845 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet b2dccb915fad4666bd7978f8c416db44
I20260317 21:19:01.980813 7791 ts_tablet_manager.cc:1916] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:01.987063 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f280d2394ccd432ea164002a2b8ce9f4" } new_table_name: "default.Table_e089a694331c4453b9ab3e9643c7b9fb" modify_external_catalogs: true
I20260317 21:19:01.991657 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_0d5bd4305663413496306f96a3b22926"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:01.961476 7882 tablet.cc:1723] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260317 21:19:01.993335 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0d5bd4305663413496306f96a3b22926 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:01.996344 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 88eae1e70bf94ef3ae0aa64706dcb599
I20260317 21:19:01.999300 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_67179be887be4140976c9d859cba3a88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.000888 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_67179be887be4140976c9d859cba3a88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:01.997534 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_245e4522696f49ee8303db7daf98929f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.003233 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_245e4522696f49ee8303db7daf98929f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.006780 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 88eae1e70bf94ef3ae0aa64706dcb599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:01 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:02.016170 7462 tablet_bootstrap.cc:492] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:02.016665 7462 ts_tablet_manager.cc:1403] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.259s user 0.011s sys 0.004s
I20260317 21:19:02.019165 7462 raft_consensus.cc:359] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.020068 7462 raft_consensus.cc:385] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:02.020337 7462 raft_consensus.cc:740] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:02.016814 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet b2dccb915fad4666bd7978f8c416db44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:01 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:02.020984 7462 consensus_queue.cc:260] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.021836 7264 leader_election.cc:304] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:02.024856 7459 tablet_bootstrap.cc:492] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:02.038339 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 70079.6us
I20260317 21:19:02.044620 7878 raft_consensus.cc:2749] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:02.047178 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 135069us
W20260317 21:19:02.048243 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 039d8bed256a43a6b98075b8430cbf3b consensus state has no leader
I20260317 21:19:02.058862 7459 tablet_bootstrap.cc:654] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:02.068425 7157 tablet_service.cc:1558] Processing DeleteTablet for tablet 88eae1e70bf94ef3ae0aa64706dcb599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:01 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:02.072685 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 35023195b9094d58827921a1dcb2ff05 because the tablet was not found. No further retry: Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05
I20260317 21:19:02.075641 7164 tablet_service.cc:1558] Processing DeleteTablet for tablet e858a01bc4594cc58853ea3239a8dbad with delete_type TABLET_DATA_DELETED (replaced by tablet b9d1076839894509805be59b751aa847) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.086746 7467 raft_consensus.cc:359] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.087456 7467 raft_consensus.cc:385] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:02.087708 7467 raft_consensus.cc:740] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:02.069587 7717 raft_consensus.cc:493] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:02.088999 7717 raft_consensus.cc:515] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.069103 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet b2dccb915fad4666bd7978f8c416db44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:01 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.092839 7717 leader_election.cc:290] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:02.088302 7467 consensus_queue.cc:260] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.109802 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet a4cee7952cf5428b98340789a48d2da2 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:02.116112 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "935faf7f9129440a91635bfa96191cd6" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:02.118052 7120 leader_election.cc:343] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:02.120203 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 50350.1us
I20260317 21:19:02.127458 7878 raft_consensus.cc:493] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:02.128062 7878 raft_consensus.cc:515] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:02.129289 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 88eae1e70bf94ef3ae0aa64706dcb599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eae1e70bf94ef3ae0aa64706dcb599 already in progress: creating tablet
I20260317 21:19:02.130894 7878 leader_election.cc:290] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:02.133237 7462 ts_tablet_manager.cc:1434] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.116s user 0.007s sys 0.000s
I20260317 21:19:02.137152 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_2f865cde34f646c6b0058233b3bbc3a6" } new_table_name: "default.Table_6157fa5cda0e4c34817e01c6bdde4904" modify_external_catalogs: true
I20260317 21:19:02.138209 7467 ts_tablet_manager.cc:1434] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.236s user 0.006s sys 0.001s
I20260317 21:19:02.139271 7467 tablet_bootstrap.cc:492] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:02.144908 7869 tablet_replica.cc:333] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:02.145427 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:02.145761 7869 raft_consensus.cc:2243] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:02.146229 7869 raft_consensus.cc:2272] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:02.135114 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet b2dccb915fad4666bd7978f8c416db44 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:01 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.135969 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet 88eae1e70bf94ef3ae0aa64706dcb599 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:01 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:19:02.147105 7261 leader_election.cc:343] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:02.151381 7878 raft_consensus.cc:493] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:02.158043 7878 raft_consensus.cc:515] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.160928 7878 leader_election.cc:290] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:02.136505 7264 leader_election.cc:343] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:02.161739 7264 leader_election.cc:304] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:02.162954 7264 consensus_peers.cc:597] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:02.171535 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_579fde97e6404c37adf7e33736201ea4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.173430 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_579fde97e6404c37adf7e33736201ea4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.174582 7459 tablet_bootstrap.cc:492] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:02.175006 7459 ts_tablet_manager.cc:1403] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.008s
I20260317 21:19:02.163437 7878 raft_consensus.cc:2749] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:02.169531 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet e858a01bc4594cc58853ea3239a8dbad with delete_type TABLET_DATA_DELETED (replaced by tablet b9d1076839894509805be59b751aa847) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.178224 7791 ts_tablet_manager.cc:1929] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:02.178623 7791 log.cc:1199] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/8ba0874a8cd746e9a33596ea252b3b14
I20260317 21:19:02.179188 7866 tablet_replica.cc:333] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:02.179312 7791 ts_tablet_manager.cc:1950] T 8ba0874a8cd746e9a33596ea252b3b14 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:02.180050 7866 raft_consensus.cc:2243] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:02.180670 7866 raft_consensus.cc:2272] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:02.181221 7459 raft_consensus.cc:359] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.181886 7459 raft_consensus.cc:385] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:02.182130 7459 raft_consensus.cc:740] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:02.182687 7866 ts_tablet_manager.cc:1916] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:02.170647 7462 tablet_bootstrap.cc:492] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:02.201970 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]). Expected version 1 got 0
I20260317 21:19:02.203691 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_c846e042b2d4448ab3aa5e50bda65ec4" } modify_external_catalogs: true
I20260317 21:19:02.204315 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_c846e042b2d4448ab3aa5e50bda65ec4" } modify_external_catalogs: true
I20260317 21:19:02.206198 7015 tablet_service.cc:1511] Processing CreateTablet for tablet a5207adefc074c229e4020f5c9406c59 (DEFAULT_TABLE table=default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.207438 6666 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ba79dc7832c24e2c980af6335f16d998" } modify_external_catalogs: true
I20260317 21:19:02.207305 7459 consensus_queue.cc:260] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.207928 6666 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ba79dc7832c24e2c980af6335f16d998" } modify_external_catalogs: true
I20260317 21:19:02.208458 7887 raft_consensus.cc:493] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:02.180127 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 132) took 2691 ms. Trace:
I20260317 21:19:02.213387 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]). Expected version 1 got 0
I20260317 21:19:02.217331 6668 catalog_manager.cc:5671] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67). New cstate: current_term: 1 leader_uuid: "2f539d1a4a584b8ca83928be49369bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } }
I20260317 21:19:02.218680 6668 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]). Expected version 1 got 0
I20260317 21:19:02.219468 6668 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_66c25b9b83724e718f148f260ec5f672 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 2 got 0
I20260317 21:19:02.219998 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5207adefc074c229e4020f5c9406c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.221673 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b287c83b77548f19546927b05905746" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:02.223659 7869 ts_tablet_manager.cc:1916] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:02.228520 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:02.230934 7261 leader_election.cc:343] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:02.231835 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet b2dccb915fad4666bd7978f8c416db44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2dccb915fad4666bd7978f8c416db44 already in progress: creating tablet
W20260317 21:19:02.232481 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 88eae1e70bf94ef3ae0aa64706dcb599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eae1e70bf94ef3ae0aa64706dcb599 already in progress: creating tablet
I20260317 21:19:02.232889 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b287c83b77548f19546927b05905746" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:02.260017 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bd7dbccbd72148e7ae945d5fded9e16a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:02.260911 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_6bb2d165f08a40d7bd84414364358f7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.261575 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bd7dbccbd72148e7ae945d5fded9e16a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:02.262248 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6bb2d165f08a40d7bd84414364358f7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.264127 6670 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_24103dddfd444503aa58762e5d598637" } new_table_name: "default.Table_d4e37f648814488a8969d384ba941f06" modify_external_catalogs: true
I20260317 21:19:02.269735 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e3e7da052b2e491e8471865daead13c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.271441 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e3e7da052b2e491e8471865daead13c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.275038 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet a4cee7952cf5428b98340789a48d2da2
I20260317 21:19:02.218988 7791 rpcz_store.cc:276] 0317 21:18:59.488321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.601750 (+113429us) service_pool.cc:225] Handling call
0317 21:19:01.929770 (+2328020us) ts_tablet_manager.cc:1151] Deleting tablet 8ba0874a8cd746e9a33596ea252b3b14
0317 21:19:01.929819 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:02.110380 (+180561us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:02.177969 (+ 67589us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:02.180071 (+ 2102us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2276445}
I20260317 21:19:02.276686 6667 catalog_manager.cc:6741] ReplaceTablet: tablet a4cee7952cf5428b98340789a48d2da2 deleted and replaced by tablet 829bc3959a9e4c2dbb6277cde5969b0e
I20260317 21:19:02.276496 7791 tablet_replica.cc:333] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:02.277801 7791 raft_consensus.cc:2243] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:02.278159 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 8ba0874a8cd746e9a33596ea252b3b14 (table default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]) successfully deleted
I20260317 21:19:02.278419 7791 raft_consensus.cc:2272] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:02.279662 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet a4cee7952cf5428b98340789a48d2da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 829bc3959a9e4c2dbb6277cde5969b0e) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:02.213670 7459 ts_tablet_manager.cc:1434] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.038s user 0.007s sys 0.000s
I20260317 21:19:02.283715 7459 tablet_bootstrap.cc:492] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:02.295255 7467 tablet_bootstrap.cc:654] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:02.208992 7887 raft_consensus.cc:515] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.323038 7152 tablet_service.cc:1511] Processing CreateTablet for tablet a5207adefc074c229e4020f5c9406c59 (DEFAULT_TABLE table=default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.324734 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5207adefc074c229e4020f5c9406c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.311527 7887 leader_election.cc:290] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:02.347270 7119 leader_election.cc:343] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 935faf7f9129440a91635bfa96191cd6
I20260317 21:19:02.347689 7119 leader_election.cc:304] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:02.348343 7887 raft_consensus.cc:2749] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:02.350158 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2f7c1058f8245428f38fe3245621b75" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:02.351953 7264 leader_election.cc:343] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:02.353150 7150 tablet_service.cc:1511] Processing CreateTablet for tablet bbc7e853c2474ba9bc6981601fdc820f (DEFAULT_TABLE table=default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.354220 6668 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]). Expected version 1 got 0
I20260317 21:19:02.354594 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc7e853c2474ba9bc6981601fdc820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.355401 6668 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:02.360975 7161 tablet_service.cc:1558] Processing DeleteTablet for tablet a4cee7952cf5428b98340789a48d2da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 829bc3959a9e4c2dbb6277cde5969b0e) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.362022 7867 raft_consensus.cc:493] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:02.362603 7867 raft_consensus.cc:515] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.368831 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82f8dd260a134810a50465e4a77848ca" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:02.369532 7184 raft_consensus.cc:2468] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:02.368403 7867 leader_election.cc:290] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:02.372161 6971 leader_election.cc:304] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:02.373544 7867 raft_consensus.cc:2804] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20260317 21:19:02.373824 6973 leader_election.cc:343] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 82f8dd260a134810a50465e4a77848ca
I20260317 21:19:02.374099 7867 raft_consensus.cc:493] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:02.374548 7867 raft_consensus.cc:3060] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:02.377506 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_713bafc801d244d1a80d9157aebd581b" } new_table_name: "default.Table_dad8a2efd2824c4e81314759e5700d97" modify_external_catalogs: true
I20260317 21:19:02.383222 7016 tablet_service.cc:1511] Processing CreateTablet for tablet bbc7e853c2474ba9bc6981601fdc820f (DEFAULT_TABLE table=default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.384757 7012 tablet_service.cc:1511] Processing CreateTablet for tablet f427b15a7b2a47e18271e8884c637f4e (DEFAULT_TABLE table=default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:02.401723 7120 leader_election.cc:343] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:02.402276 6669 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6] alter complete (version 1)
W20260317 21:19:02.407339 6673 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:45181 (request call id 46) took 977 ms (client timeout 1000 ms). Trace:
W20260317 21:19:02.408414 6673 rpcz_store.cc:269] 0317 21:19:01.429853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:01.431771 (+ 1918us) service_pool.cc:225] Handling call
0317 21:19:02.407277 (+975506us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10516,"reported_tablets":5}
I20260317 21:19:02.411520 7467 tablet_bootstrap.cc:492] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:02.411986 7467 ts_tablet_manager.cc:1403] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.273s user 0.006s sys 0.007s
I20260317 21:19:02.417977 7867 raft_consensus.cc:515] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.421121 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "82f8dd260a134810a50465e4a77848ca" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:02.421701 7184 raft_consensus.cc:3060] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:02.425671 7161 tablet_service.cc:1511] Processing CreateTablet for tablet f427b15a7b2a47e18271e8884c637f4e (DEFAULT_TABLE table=default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.427861 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f427b15a7b2a47e18271e8884c637f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.429452 7184 raft_consensus.cc:2468] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:02.351783 7302 tablet_service.cc:1511] Processing CreateTablet for tablet a5207adefc074c229e4020f5c9406c59 (DEFAULT_TABLE table=default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.433626 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5207adefc074c229e4020f5c9406c59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.435473 6668 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]). Expected version 1 got 0
I20260317 21:19:02.352573 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet a4cee7952cf5428b98340789a48d2da2 with delete_type TABLET_DATA_DELETED (replaced by tablet 829bc3959a9e4c2dbb6277cde5969b0e) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.353175 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.430981 7462 tablet_bootstrap.cc:654] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:02.440788 7264 leader_election.cc:304] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:02.441558 7878 raft_consensus.cc:2749] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:02.441622 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 89701.6us
I20260317 21:19:02.446189 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b9d303f100ef4742bab4a7a0a7c97a1c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.447690 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b9d303f100ef4742bab4a7a0a7c97a1c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:02.451789 7264 leader_election.cc:343] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:02.453058 7264 leader_election.cc:304] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:02.454383 7878 raft_consensus.cc:2749] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:02.458422 7301 tablet_service.cc:1511] Processing CreateTablet for tablet bbc7e853c2474ba9bc6981601fdc820f (DEFAULT_TABLE table=default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.458665 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.459746 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc7e853c2474ba9bc6981601fdc820f. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:02.459828 7119 leader_election.cc:343] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: d2f7c1058f8245428f38fe3245621b75
I20260317 21:19:02.460320 7119 leader_election.cc:304] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:02.462139 7300 tablet_service.cc:1511] Processing CreateTablet for tablet f427b15a7b2a47e18271e8884c637f4e (DEFAULT_TABLE table=default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.463500 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f427b15a7b2a47e18271e8884c637f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.476941 7887 raft_consensus.cc:2749] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:02.490841 7462 tablet_bootstrap.cc:492] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:02.491449 7462 ts_tablet_manager.cc:1403] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.321s user 0.009s sys 0.004s
I20260317 21:19:02.494374 7462 raft_consensus.cc:359] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.495131 7462 raft_consensus.cc:385] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:02.495440 7462 raft_consensus.cc:740] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:02.496910 7467 raft_consensus.cc:359] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.506418 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 7d73f093d0e8434ca4c41cef6dfe310c
I20260317 21:19:02.509678 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 252f728d183f440a9ef42894508d5755
I20260317 21:19:02.511497 7866 ts_tablet_manager.cc:1929] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:02.511996 7866 log.cc:1199] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/e858a01bc4594cc58853ea3239a8dbad
I20260317 21:19:02.513388 6971 leader_election.cc:304] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:02.515103 6670 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e4522181cc434203ac14c9b07587fec8 consensus state has no leader
I20260317 21:19:02.517581 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d73f093d0e8434ca4c41cef6dfe310c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.519491 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 252f728d183f440a9ef42894508d5755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.553946 7904 raft_consensus.cc:2804] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:02.563522 7462 consensus_queue.cc:260] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.572995 7867 leader_election.cc:290] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:02.574800 6662 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet e4522181cc434203ac14c9b07587fec8 consensus state has no leader
I20260317 21:19:02.576844 7462 ts_tablet_manager.cc:1434] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.085s user 0.006s sys 0.000s
W20260317 21:19:02.576257 6973 leader_election.cc:343] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 82f8dd260a134810a50465e4a77848ca
I20260317 21:19:02.594594 6672 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
W20260317 21:19:02.596736 6672 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9b6275af1c9c41ae8e4515deabdbdee1 consensus state has no leader
I20260317 21:19:02.617487 6672 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:02.619520 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbc7e853c2474ba9bc6981601fdc820f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.624385 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f427b15a7b2a47e18271e8884c637f4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.627946 7462 tablet_bootstrap.cc:492] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:02.664347 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 151072us
I20260317 21:19:02.617930 7791 ts_tablet_manager.cc:1916] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:02.679034 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 70560dde990843e0b7c74ae3206c58e4 (DEFAULT_TABLE table=default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.501019 7163 tablet_service.cc:1558] Processing DeleteTablet for tablet 82f8dd260a134810a50465e4a77848ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd3e35e25bf444cbd821ebd9f5ee018) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.502089 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 935faf7f9129440a91635bfa96191cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:56 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.564814 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet d2f7c1058f8245428f38fe3245621b75 with delete_type TABLET_DATA_DELETED (replaced by tablet e22772281bd74754bb969e87a70d8077) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.567627 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d73f093d0e8434ca4c41cef6dfe310c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.685216 7467 raft_consensus.cc:385] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:02.568460 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 252f728d183f440a9ef42894508d5755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.685608 7467 raft_consensus.cc:740] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:02.581017 7905 tablet.cc:1723] T b5a797b22cf34483b0061955da6b2634 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:02.686576 7467 consensus_queue.cc:260] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.680263 7907 tablet.cc:1723] T b5a797b22cf34483b0061955da6b2634 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:02.694835 7459 tablet_bootstrap.cc:654] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:02.700140 6674 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]). Expected version 1 got 0
I20260317 21:19:02.704774 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70560dde990843e0b7c74ae3206c58e4. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:02.710875 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fd7cf19121b5496a849118de4a29db5b consensus state has no leader
I20260317 21:19:02.731762 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4088537e0de64ecc8cba0ebff9c50bbc" } modify_external_catalogs: true
I20260317 21:19:02.732441 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4088537e0de64ecc8cba0ebff9c50bbc" } modify_external_catalogs: true
I20260317 21:19:02.734035 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_9f4465b6fdf24369a96e3bf37cfd19d7" } new_table_name: "default.Table_76a974b216ba482e9783ae7f126c5d31" modify_external_catalogs: true
I20260317 21:19:02.741294 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 7d73f093d0e8434ca4c41cef6dfe310c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:02.752420 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 252f728d183f440a9ef42894508d5755 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:02.752822 7462 tablet_bootstrap.cc:700] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Time spent opening tablet: real 0.120s user 0.004s sys 0.000s
I20260317 21:19:02.754173 7462 tablet_bootstrap.cc:654] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:02.769299 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_687dd24cc64a425b80bcfb73ef08d1fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.770953 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_687dd24cc64a425b80bcfb73ef08d1fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.770993 7869 ts_tablet_manager.cc:1929] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:02.771365 7869 log.cc:1199] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/e858a01bc4594cc58853ea3239a8dbad
I20260317 21:19:02.776041 6672 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ef1aaeadf5354f7e99d9db422dfc6e63" } modify_external_catalogs: true
I20260317 21:19:02.776805 6672 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ef1aaeadf5354f7e99d9db422dfc6e63" } modify_external_catalogs: true
I20260317 21:19:02.803308 7866 ts_tablet_manager.cc:1950] T e858a01bc4594cc58853ea3239a8dbad P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:02.812541 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b78a8f62954143d9b88c4fa88d44cbf2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.814172 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b78a8f62954143d9b88c4fa88d44cbf2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.827905 7155 tablet_service.cc:1558] Processing DeleteTablet for tablet dc929bae186841d2aba674659e457301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:02.828876 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:02.823678 7303 tablet_service.cc:1511] Processing CreateTablet for tablet d4afbb1ab93a44d9ab5795b72444c690 (DEFAULT_TABLE table=default.Table_f95f849f2a4340bbb5f515633e6f50fe [id=3a32714ce85b4daaa1203f724d4b52e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.833606 7467 ts_tablet_manager.cc:1434] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.421s user 0.006s sys 0.003s
I20260317 21:19:02.834100 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 70560dde990843e0b7c74ae3206c58e4 (DEFAULT_TABLE table=default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.835843 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70560dde990843e0b7c74ae3206c58e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.839125 7869 ts_tablet_manager.cc:1950] T e858a01bc4594cc58853ea3239a8dbad P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:02.849398 7467 tablet_bootstrap.cc:492] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:02.865573 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_564099aea7ad48349bd3df5beacc7414"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:02.867312 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_564099aea7ad48349bd3df5beacc7414 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:02.869506 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 70560dde990843e0b7c74ae3206c58e4 (DEFAULT_TABLE table=default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.870585 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70560dde990843e0b7c74ae3206c58e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.888067 6671 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_6157fa5cda0e4c34817e01c6bdde4904 [id=4a69de09e4be4630994811c7e432c315] alter complete (version 1)
I20260317 21:19:02.906796 7006 tablet_service.cc:1511] Processing CreateTablet for tablet d4afbb1ab93a44d9ab5795b72444c690 (DEFAULT_TABLE table=default.Table_f95f849f2a4340bbb5f515633e6f50fe [id=3a32714ce85b4daaa1203f724d4b52e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.907925 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4afbb1ab93a44d9ab5795b72444c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.909359 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet a978fa28a31b4d589031c77526b62b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:02.917877 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet a978fa28a31b4d589031c77526b62b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:02.922222 7878 raft_consensus.cc:493] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:02.923171 7878 raft_consensus.cc:515] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:02.924993 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a44a297079347bab6f8be6abe332bfa consensus state has no leader
I20260317 21:19:02.924950 7878 leader_election.cc:290] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:02.928323 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b9d303f100ef4742bab4a7a0a7c97a1c" } new_table_name: "default.Table_72df81c9b06c4ac9af9af25d9862df96" modify_external_catalogs: true
I20260317 21:19:02.935992 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:02.938073 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c (DEFAULT_TABLE table=default.Table_bdd5608a590847cf98a367c051f62b45 [id=70d2f8f2445b4fab82ec7794c49c553f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:02.938725 7264 leader_election.cc:343] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:02.941489 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3beb9232d74e45ab713ae8241eec4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:02.945737 6973 consensus_peers.cc:597] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e858a01bc4594cc58853ea3239a8dbad. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:02.949241 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c (DEFAULT_TABLE table=default.Table_bdd5608a590847cf98a367c051f62b45 [id=70d2f8f2445b4fab82ec7794c49c553f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:02.954524 7462 tablet_bootstrap.cc:492] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:02.954967 7462 ts_tablet_manager.cc:1403] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.369s user 0.009s sys 0.001s
I20260317 21:19:02.957662 7462 raft_consensus.cc:359] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.958385 7462 raft_consensus.cc:385] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:02.958693 7462 raft_consensus.cc:740] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:02.959416 7462 consensus_queue.cc:260] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:02.962139 7462 ts_tablet_manager.cc:1434] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:02.963332 7462 tablet_bootstrap.cc:492] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:02.964586 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3beb9232d74e45ab713ae8241eec4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:02.973268 7462 tablet_bootstrap.cc:654] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:02.976963 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4afbb1ab93a44d9ab5795b72444c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.016173 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:03.017984 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.019989 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 7fc38947219047b5abdf71af5554a987
I20260317 21:19:03.026091 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fc38947219047b5abdf71af5554a987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:03.029335 7462 tablet_bootstrap.cc:492] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:03.029943 7462 ts_tablet_manager.cc:1403] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.067s user 0.017s sys 0.041s
I20260317 21:19:03.032842 7462 raft_consensus.cc:359] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.033804 7462 raft_consensus.cc:385] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.034161 7462 raft_consensus.cc:740] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.035115 7462 consensus_queue.cc:260] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.039630 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0b009e9d95134391a1d2d41e80db86c7" } modify_external_catalogs: true
I20260317 21:19:03.040199 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0b009e9d95134391a1d2d41e80db86c7" } modify_external_catalogs: true
I20260317 21:19:03.043015 7462 ts_tablet_manager.cc:1434] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260317 21:19:03.048466 7462 tablet_bootstrap.cc:492] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:03.057116 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 2607b9978c32466eb6dc8b8acfb429ce (DEFAULT_TABLE table=default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.058231 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2607b9978c32466eb6dc8b8acfb429ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.059314 7866 tablet_replica.cc:333] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:03.060197 7866 raft_consensus.cc:2243] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:03.067163 7866 raft_consensus.cc:2272] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.072299 7462 tablet_bootstrap.cc:654] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.075467 7866 ts_tablet_manager.cc:1916] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:03.078186 6662 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 61235.1us
I20260317 21:19:03.081049 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet e858a01bc4594cc58853ea3239a8dbad (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]) successfully deleted
W20260317 21:19:03.081689 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet a4cee7952cf5428b98340789a48d2da2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4cee7952cf5428b98340789a48d2da2 already in progress: creating tablet
W20260317 21:19:03.084591 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae1b2a549af6485ca3faee3f0ad845a2 has no consensus state
I20260317 21:19:02.881414 7904 raft_consensus.cc:697] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:03.128942 7903 raft_consensus.cc:493] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.129429 7903 raft_consensus.cc:515] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.131441 7903 leader_election.cc:290] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:03.133909 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:03.121268 7867 raft_consensus.cc:493] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader c0ef92fef382401a9564c17e0c223f6a)
I20260317 21:19:03.137413 7867 raft_consensus.cc:515] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.139454 7867 leader_election.cc:290] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:03.142271 6973 leader_election.cc:343] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:03.142966 7035 raft_consensus.cc:3060] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:03.149888 7015 consensus_meta.cc:287] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent flushing consensus metadata: real 0.602s user 0.004s sys 0.003s
W20260317 21:19:03.152894 7150 consensus_meta.cc:287] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a: Time spent flushing consensus metadata: real 0.758s user 0.003s sys 0.004s
W20260317 21:19:03.157666 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 329590us
I20260317 21:19:03.158174 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 2607b9978c32466eb6dc8b8acfb429ce (DEFAULT_TABLE table=default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.159245 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2607b9978c32466eb6dc8b8acfb429ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.159106 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 8a44a297079347bab6f8be6abe332bfa (DEFAULT_TABLE table=default.Table_76a974b216ba482e9783ae7f126c5d31 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.160028 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a44a297079347bab6f8be6abe332bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.165392 7009 tablet_service.cc:1511] Processing CreateTablet for tablet bc2cd63423f844aea29ed3fd839f083e (DEFAULT_TABLE table=default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.166466 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2cd63423f844aea29ed3fd839f083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.166324 7300 tablet_service.cc:1511] Processing CreateTablet for tablet bc2cd63423f844aea29ed3fd839f083e (DEFAULT_TABLE table=default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.167440 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2cd63423f844aea29ed3fd839f083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.167990 7904 consensus_queue.cc:237] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.172991 7459 tablet_bootstrap.cc:492] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:03.173415 7459 ts_tablet_manager.cc:1403] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.890s user 0.004s sys 0.017s
I20260317 21:19:03.175726 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fc38947219047b5abdf71af5554a987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:03.185441 7152 consensus_meta.cc:287] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a: Time spent flushing consensus metadata: real 0.507s user 0.008s sys 0.001s
I20260317 21:19:03.185124 7459 raft_consensus.cc:359] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.185819 7459 raft_consensus.cc:385] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.186112 7459 raft_consensus.cc:740] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.186826 7459 consensus_queue.cc:260] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.188467 7459 ts_tablet_manager.cc:1434] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
W20260317 21:19:03.192860 7161 consensus_meta.cc:287] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a: Time spent flushing consensus metadata: real 0.517s user 0.004s sys 0.003s
I20260317 21:19:03.193511 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.195178 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.196911 6971 leader_election.cc:304] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:03.209261 7936 raft_consensus.cc:493] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.222285 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_409b88634b48453daacc460109b51caf" } new_table_name: "default.Table_5ebd4f0e4c284b4caac2e5f0cddf4e07" modify_external_catalogs: true
I20260317 21:19:03.224723 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1ce0835b11b44dcc9535d7cf85dead34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:03.225063 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_47f5a50f046e45fb93865004ab8817c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:03.226380 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1ce0835b11b44dcc9535d7cf85dead34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:03.230019 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_47f5a50f046e45fb93865004ab8817c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.230979 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:03.237716 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 8a44a297079347bab6f8be6abe332bfa (DEFAULT_TABLE table=default.Table_76a974b216ba482e9783ae7f126c5d31 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.238984 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a44a297079347bab6f8be6abe332bfa. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:03.245278 7261 leader_election.cc:343] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.245926 7261 leader_election.cc:304] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.247766 7878 raft_consensus.cc:2749] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.258344 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 16f64073188f4d06859e1a701df73242 (DEFAULT_TABLE table=default.Table_17cf4ddfbd3d47ba99971456fc3873f6 [id=75f6dded5e5e433aa85b47d7f58b53d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.260869 7459 tablet_bootstrap.cc:492] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:03.268085 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 788f0ee1f7de47a49c5601c9b1494885 (DEFAULT_TABLE table=default.Table_a3070bf38c304436b100f2ddb8a4716f [id=df5c9c1007aa4e84aa9363779d23864b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.282524 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 2562cc2a3bb2430981c88475f9e1daa1
W20260317 21:19:03.283020 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 86222.9us
W20260317 21:19:03.286021 6971 consensus_peers.cc:597] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e858a01bc4594cc58853ea3239a8dbad. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:03.286324 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 2562cc2a3bb2430981c88475f9e1daa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:03.286827 6971 leader_election.cc:343] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:03.287294 6971 leader_election.cc:304] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:03.289135 7938 raft_consensus.cc:2749] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:03.294631 7791 ts_tablet_manager.cc:1929] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:03.295109 7791 log.cc:1199] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/53dd9f8a1ccc47f884bfba99781aed16
I20260317 21:19:03.295873 7791 ts_tablet_manager.cc:1950] T 53dd9f8a1ccc47f884bfba99781aed16 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:03.300112 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788f0ee1f7de47a49c5601c9b1494885. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:03.300151 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 77604.2us
I20260317 21:19:03.302187 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f64073188f4d06859e1a701df73242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.302587 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 141) took 3446 ms. Trace:
I20260317 21:19:03.303186 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 16f64073188f4d06859e1a701df73242 (DEFAULT_TABLE table=default.Table_17cf4ddfbd3d47ba99971456fc3873f6 [id=75f6dded5e5e433aa85b47d7f58b53d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.302999 7791 rpcz_store.cc:276] 0317 21:18:59.856297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.856408 (+ 111us) service_pool.cc:225] Handling call
0317 21:19:02.276287 (+2419879us) ts_tablet_manager.cc:1151] Deleting tablet 53dd9f8a1ccc47f884bfba99781aed16
0317 21:19:02.276348 (+ 61us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:02.901090 (+624742us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.294375 (+393285us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.302519 (+ 8144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2319096}
I20260317 21:19:03.304248 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f64073188f4d06859e1a701df73242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.305620 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 53dd9f8a1ccc47f884bfba99781aed16 (table default.Table_afb9c6074ee54f86a4d029cf8cdbb825 [id=dd080d09b8324bcabbe022d42352fadb]) successfully deleted
I20260317 21:19:03.308840 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 144) took 3447 ms. Trace:
I20260317 21:19:03.309285 7791 rpcz_store.cc:276] 0317 21:18:59.861305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.861500 (+ 195us) service_pool.cc:225] Handling call
0317 21:19:03.304924 (+3443424us) ts_tablet_manager.cc:1151] Deleting tablet 59fce28afbeb4d8eadf63f9a97d1183d
0317 21:19:03.304969 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.308766 (+ 3797us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3344068}
W20260317 21:19:03.310729 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 59fce28afbeb4d8eadf63f9a97d1183d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59fce28afbeb4d8eadf63f9a97d1183d already in progress: creating tablet
I20260317 21:19:03.314812 7462 tablet_bootstrap.cc:492] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:03.315250 7462 ts_tablet_manager.cc:1403] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.271s user 0.054s sys 0.064s
I20260317 21:19:03.317481 7462 raft_consensus.cc:359] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.318086 7462 raft_consensus.cc:385] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.318322 7462 raft_consensus.cc:740] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:19:03.231510 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 72816.2us
I20260317 21:19:03.236152 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 183) took 1301 ms. Trace:
I20260317 21:19:03.323411 7869 rpcz_store.cc:276] 0317 21:19:01.934648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.037810 (+103162us) service_pool.cc:225] Handling call
0317 21:19:02.128092 (+ 90282us) ts_tablet_manager.cc:1151] Deleting tablet e858a01bc4594cc58853ea3239a8dbad
0317 21:19:02.128130 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:02.397212 (+269082us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:02.770721 (+373509us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.235796 (+465075us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":388270,"tablet-delete.queue_time_us":51902}
I20260317 21:19:03.318895 7462 consensus_queue.cc:260] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.325430 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 184) took 1286 ms. Trace:
I20260317 21:19:03.325788 7869 rpcz_store.cc:276] 0317 21:19:02.039122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.068952 (+ 29830us) service_pool.cc:225] Handling call
0317 21:19:03.324889 (+1255937us) ts_tablet_manager.cc:1151] Deleting tablet b2dccb915fad4666bd7978f8c416db44
0317 21:19:03.324991 (+ 102us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.325371 (+ 380us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1233753}
I20260317 21:19:03.328878 7869 tablet_replica.cc:333] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:03.329687 7869 raft_consensus.cc:2243] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:03.330258 7869 raft_consensus.cc:2272] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.236145 7467 tablet_bootstrap.cc:654] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.335103 7462 ts_tablet_manager.cc:1434] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260317 21:19:03.336057 7462 tablet_bootstrap.cc:492] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:03.341825 7462 tablet_bootstrap.cc:654] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.342087 7162 tablet_service.cc:1511] Processing CreateTablet for tablet d4afbb1ab93a44d9ab5795b72444c690 (DEFAULT_TABLE table=default.Table_f95f849f2a4340bbb5f515633e6f50fe [id=3a32714ce85b4daaa1203f724d4b52e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.343722 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d4afbb1ab93a44d9ab5795b72444c690. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.340517 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet e858a01bc4594cc58853ea3239a8dbad (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]) successfully deleted
I20260317 21:19:03.344765 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c (DEFAULT_TABLE table=default.Table_bdd5608a590847cf98a367c051f62b45 [id=70d2f8f2445b4fab82ec7794c49c553f]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:03.344466 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet b2dccb915fad4666bd7978f8c416db44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2dccb915fad4666bd7978f8c416db44 already in progress: creating tablet
I20260317 21:19:03.345785 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c3beb9232d74e45ab713ae8241eec4c. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:03.345791 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet a4cee7952cf5428b98340789a48d2da2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4cee7952cf5428b98340789a48d2da2 already in progress: creating tablet
I20260317 21:19:03.355310 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 2562cc2a3bb2430981c88475f9e1daa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:03.359678 7791 tablet_replica.cc:333] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:03.360692 7791 raft_consensus.cc:2243] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:03.361866 7791 pending_rounds.cc:70] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:03.362051 7462 tablet_bootstrap.cc:492] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:03.362670 7462 ts_tablet_manager.cc:1403] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.001s
I20260317 21:19:03.367522 7462 raft_consensus.cc:359] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.368211 7462 raft_consensus.cc:385] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.369233 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ce47367ed77d4c06ad8c624c31dd3b2b" } modify_external_catalogs: true
I20260317 21:19:03.369781 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_ce47367ed77d4c06ad8c624c31dd3b2b" } modify_external_catalogs: true
I20260317 21:19:03.368469 7462 raft_consensus.cc:740] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.372323 7462 consensus_queue.cc:260] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.374935 7462 ts_tablet_manager.cc:1434] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.012s user 0.005s sys 0.000s
I20260317 21:19:03.375896 7462 tablet_bootstrap.cc:492] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:03.378459 6674 master_service.cc:954] ReplaceTablet: received request to replace tablet 35023195b9094d58827921a1dcb2ff05 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:03.380715 7906 raft_consensus.cc:493] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.384256 7906 raft_consensus.cc:515] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:03.388926 6973 leader_election.cc:343] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: c6d5385171364a078d968bb0a489383a
I20260317 21:19:03.389161 7462 tablet_bootstrap.cc:654] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.389778 7906 leader_election.cc:290] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:03.389750 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 788f0ee1f7de47a49c5601c9b1494885 (DEFAULT_TABLE table=default.Table_a3070bf38c304436b100f2ddb8a4716f [id=df5c9c1007aa4e84aa9363779d23864b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.391005 7791 pending_rounds.cc:77] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412459416838144 op_type: NO_OP noop_request { }
I20260317 21:19:03.396483 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_6028353a5dad482db231700fb6461738"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:03.398193 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6028353a5dad482db231700fb6461738 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.397042 7791 raft_consensus.cc:2889] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:03.401360 7791 raft_consensus.cc:2272] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.401449 6671 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_a3070bf38c304436b100f2ddb8a4716f" } new_table_name: "default.Table_a33d113c213c48728df41846ad097ead" modify_external_catalogs: true
I20260317 21:19:03.401422 7008 tablet_service.cc:1511] Processing CreateTablet for tablet b76fddff23d04a9a91ea26a366249784 (DEFAULT_TABLE table=default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.402853 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b76fddff23d04a9a91ea26a366249784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.409821 7462 tablet_bootstrap.cc:492] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:03.410315 7462 ts_tablet_manager.cc:1403] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.035s user 0.009s sys 0.007s
I20260317 21:19:03.412869 7462 raft_consensus.cc:359] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.413528 7462 raft_consensus.cc:385] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.413620 7878 raft_consensus.cc:493] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.413779 7462 raft_consensus.cc:740] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.414162 7878 raft_consensus.cc:515] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.414399 7462 consensus_queue.cc:260] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.416203 7878 leader_election.cc:290] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:03.418155 7462 ts_tablet_manager.cc:1434] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.007s user 0.005s sys 0.000s
I20260317 21:19:03.419322 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a978fa28a31b4d589031c77526b62b8e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:03.419925 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet 589aadc7897b422e90d3f7caed76ab59 from {username='slave'} at 127.0.0.1:39038
W20260317 21:19:03.426632 7264 leader_election.cc:343] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.427222 7878 raft_consensus.cc:493] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.429134 7459 tablet_bootstrap.cc:654] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.429003 7878 raft_consensus.cc:515] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:03.430955 6639 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 48564.7us
I20260317 21:19:03.431749 7866 ts_tablet_manager.cc:1929] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:03.431833 7878 leader_election.cc:290] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:03.433480 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:03.435163 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:03.437057 7264 leader_election.cc:343] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:03.440068 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2a221dfad830428db2329d6e543f41dc consensus state has no leader
I20260317 21:19:03.448733 7791 ts_tablet_manager.cc:1916] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:03.458431 7301 tablet_service.cc:1511] Processing CreateTablet for tablet b76fddff23d04a9a91ea26a366249784 (DEFAULT_TABLE table=default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:03.474550 6662 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111) [suppressed 7 similar messages]
I20260317 21:19:03.432178 7866 log.cc:1199] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/78e2e20d320547b3bb1e03923de52822
I20260317 21:19:03.476955 7878 raft_consensus.cc:493] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.478528 7943 raft_consensus.cc:2749] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.479029 7866 ts_tablet_manager.cc:1950] T 78e2e20d320547b3bb1e03923de52822 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:03.477587 7878 raft_consensus.cc:515] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.482692 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb332f981fc84a178fadae2f4908bdde" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:03.484200 7264 leader_election.cc:343] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.485697 7878 leader_election.cc:290] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:03.496253 7878 raft_consensus.cc:493] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.498731 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788f0ee1f7de47a49c5601c9b1494885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.500818 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 82f8dd260a134810a50465e4a77848ca with delete_type TABLET_DATA_DELETED (replaced by tablet 0fd3e35e25bf444cbd821ebd9f5ee018) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:03.505720 7462 tablet_bootstrap.cc:492] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:03.509644 7878 raft_consensus.cc:515] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.510977 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b76fddff23d04a9a91ea26a366249784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.513336 7462 tablet_bootstrap.cc:654] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.514989 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59fce28afbeb4d8eadf63f9a97d1183d" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:03.516067 7878 leader_election.cc:290] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:03.516254 7264 leader_election.cc:343] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.523877 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a978fa28a31b4d589031c77526b62b8e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.525100 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fc38947219047b5abdf71af5554a987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:03.526122 6672 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:03.526751 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb332f981fc84a178fadae2f4908bdde" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.526893 7866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 186) took 1175 ms. Trace:
I20260317 21:19:03.527223 7866 rpcz_store.cc:276] 0317 21:19:02.351705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.352982 (+ 1277us) service_pool.cc:225] Handling call
0317 21:19:03.059158 (+706176us) ts_tablet_manager.cc:1151] Deleting tablet 78e2e20d320547b3bb1e03923de52822
0317 21:19:03.059193 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.247178 (+187985us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.431473 (+184295us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.526821 (+ 95348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":622059}
I20260317 21:19:03.529055 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59fce28afbeb4d8eadf63f9a97d1183d" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.528749 7936 raft_consensus.cc:515] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.529711 7866 tablet_replica.cc:333] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:03.529865 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 78e2e20d320547b3bb1e03923de52822 (table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]) successfully deleted
W20260317 21:19:03.530534 7261 leader_election.cc:343] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.531127 7936 leader_election.cc:290] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:03.531347 7261 leader_election.cc:304] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:03.533156 7261 leader_election.cc:343] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.533572 7261 leader_election.cc:304] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.534622 7957 raft_consensus.cc:2749] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.535166 7957 raft_consensus.cc:2749] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.528528 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:03.537907 7261 leader_election.cc:343] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.538440 7261 leader_election.cc:304] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.539229 7957 raft_consensus.cc:2749] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:03.540307 7261 leader_election.cc:343] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.540421 7152 tablet_service.cc:1558] Processing DeleteTablet for tablet a978fa28a31b4d589031c77526b62b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:03.515277 7869 ts_tablet_manager.cc:1916] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:03.540859 7261 leader_election.cc:304] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.541544 7957 raft_consensus.cc:2749] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.553110 7155 tablet_service.cc:1511] Processing CreateTablet for tablet bc2cd63423f844aea29ed3fd839f083e (DEFAULT_TABLE table=default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.554370 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc2cd63423f844aea29ed3fd839f083e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.553938 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_03e967dfb6ec4350932c7c69cec172b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:03.555689 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03e967dfb6ec4350932c7c69cec172b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.555718 7866 raft_consensus.cc:2243] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:03.556689 7866 raft_consensus.cc:2272] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.557507 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 16f64073188f4d06859e1a701df73242 (DEFAULT_TABLE table=default.Table_17cf4ddfbd3d47ba99971456fc3873f6 [id=75f6dded5e5e433aa85b47d7f58b53d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.558539 7866 ts_tablet_manager.cc:1916] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:03.558969 7159 tablet_service.cc:1511] Processing CreateTablet for tablet b76fddff23d04a9a91ea26a366249784 (DEFAULT_TABLE table=default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.559938 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b76fddff23d04a9a91ea26a366249784. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.561017 7163 tablet_service.cc:1558] Processing DeleteTablet for tablet 2562cc2a3bb2430981c88475f9e1daa1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:03.562403 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 788f0ee1f7de47a49c5601c9b1494885 (DEFAULT_TABLE table=default.Table_a3070bf38c304436b100f2ddb8a4716f [id=df5c9c1007aa4e84aa9363779d23864b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.563398 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet c9b7f7c864004b9abde0a0201e196165
I20260317 21:19:03.568298 7878 raft_consensus.cc:493] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.568819 7878 raft_consensus.cc:515] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.569386 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 8a44a297079347bab6f8be6abe332bfa (DEFAULT_TABLE table=default.Table_76a974b216ba482e9783ae7f126c5d31 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:03.572664 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b066a6a5c864beb8da59bd97bacd309" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.573721 7152 tablet_service.cc:1511] Processing CreateTablet for tablet 2607b9978c32466eb6dc8b8acfb429ce (DEFAULT_TABLE table=default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:03.574179 7261 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.577069 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet c9b7f7c864004b9abde0a0201e196165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:19:03.578863 7119 leader_election.cc:343] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: dc929bae186841d2aba674659e457301
I20260317 21:19:03.578943 7878 leader_election.cc:290] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:03.580983 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16f64073188f4d06859e1a701df73242. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.581043 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bdd5608a590847cf98a367c051f62b45" } modify_external_catalogs: true
I20260317 21:19:03.581563 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bdd5608a590847cf98a367c051f62b45" } modify_external_catalogs: true
I20260317 21:19:03.585178 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 788f0ee1f7de47a49c5601c9b1494885. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.589099 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a44a297079347bab6f8be6abe332bfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.593031 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2607b9978c32466eb6dc8b8acfb429ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:03.612679 7968 tablet.cc:1723] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:03.621474 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet c6d5385171364a078d968bb0a489383a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:54 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:03.601639 6971 leader_election.cc:343] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: c6d5385171364a078d968bb0a489383a
I20260317 21:19:03.622391 7866 ts_tablet_manager.cc:1929] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:03.622807 7866 log.cc:1199] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/2b287c83b77548f19546927b05905746
I20260317 21:19:03.622573 6971 leader_election.cc:304] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:03.623464 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ae1b2a549af6485ca3faee3f0ad845a2 (table default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ae1b2a549af6485ca3faee3f0ad845a2 has no consensus state
I20260317 21:19:03.623533 7866 ts_tablet_manager.cc:1950] T 2b287c83b77548f19546927b05905746 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:03.623862 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b066a6a5c864beb8da59bd97bacd309" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:03.625298 7264 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.625697 7264 leader_election.cc:304] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.625902 7866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 189) took 1167 ms. Trace:
I20260317 21:19:03.626209 7866 rpcz_store.cc:276] 0317 21:19:02.457997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.458506 (+ 509us) service_pool.cc:225] Handling call
0317 21:19:03.529533 (+1071027us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:19:03.529591 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.574654 (+ 45063us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.608435 (+ 33781us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.625832 (+ 17397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1070285}
I20260317 21:19:03.627753 7866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 190) took 1111 ms. Trace:
I20260317 21:19:03.628118 7866 rpcz_store.cc:276] 0317 21:19:02.516737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.516898 (+ 161us) service_pool.cc:225] Handling call
0317 21:19:03.627360 (+1110462us) ts_tablet_manager.cc:1151] Deleting tablet 7d73f093d0e8434ca4c41cef6dfe310c
0317 21:19:03.627411 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.627693 (+ 282us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109312}
I20260317 21:19:03.629370 7866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 191) took 1111 ms. Trace:
I20260317 21:19:03.629662 7866 rpcz_store.cc:276] 0317 21:19:02.517911 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.519314 (+ 1403us) service_pool.cc:225] Handling call
0317 21:19:03.629013 (+1109699us) ts_tablet_manager.cc:1151] Deleting tablet 252f728d183f440a9ef42894508d5755
0317 21:19:03.629056 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.629320 (+ 264us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1109010}
I20260317 21:19:03.630728 7866 tablet_replica.cc:333] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:03.631603 7866 raft_consensus.cc:2243] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:03.632076 7866 raft_consensus.cc:2272] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.637027 7906 raft_consensus.cc:2749] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.640928 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 2b287c83b77548f19546927b05905746 (table default.Table_e4715bec3faa471cbdcb63c57766180a [id=ff96f5df47e0414290054b29ed53479a]) successfully deleted
W20260317 21:19:03.641742 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 7d73f093d0e8434ca4c41cef6dfe310c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d73f093d0e8434ca4c41cef6dfe310c already in progress: creating tablet
I20260317 21:19:03.642119 7866 ts_tablet_manager.cc:1916] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:03.642345 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 252f728d183f440a9ef42894508d5755 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252f728d183f440a9ef42894508d5755 already in progress: creating tablet
I20260317 21:19:03.655371 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1e952ee93a1d4d7abcd06a732581a9d2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:03.656960 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e952ee93a1d4d7abcd06a732581a9d2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.656701 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_044bd8a2b853439c8591bbc87743c963"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:03.657563 7957 raft_consensus.cc:2749] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:03.658463 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_044bd8a2b853439c8591bbc87743c963 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.661646 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d99f91a2b9224cb9a922e32c03228878" } new_table_name: "default.Table_38fe3ad6f1bf4796af58d99b53c46560" modify_external_catalogs: true
I20260317 21:19:03.663607 7869 ts_tablet_manager.cc:1929] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:03.664083 7869 log.cc:1199] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/82f8dd260a134810a50465e4a77848ca
I20260317 21:19:03.665002 7869 ts_tablet_manager.cc:1950] T 82f8dd260a134810a50465e4a77848ca P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:03.667291 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 192) took 1177 ms. Trace:
I20260317 21:19:03.667683 7869 rpcz_store.cc:276] 0317 21:19:02.489521 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.500843 (+ 11322us) service_pool.cc:225] Handling call
0317 21:19:03.328706 (+827863us) ts_tablet_manager.cc:1151] Deleting tablet 82f8dd260a134810a50465e4a77848ca
0317 21:19:03.328748 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.607619 (+278871us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.652961 (+ 45342us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.667226 (+ 14265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":643749}
I20260317 21:19:03.675813 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet c9b7f7c864004b9abde0a0201e196165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:03.678478 7467 tablet_bootstrap.cc:492] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:03.679018 7467 ts_tablet_manager.cc:1403] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.830s user 0.021s sys 0.026s
I20260317 21:19:03.686076 7866 ts_tablet_manager.cc:1929] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:03.686666 7866 log.cc:1199] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:03.687505 7866 ts_tablet_manager.cc:1950] T a978fa28a31b4d589031c77526b62b8e P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:03.690912 7869 tablet_replica.cc:333] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:03.690814 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet a978fa28a31b4d589031c77526b62b8e (table default.Table_4088537e0de64ecc8cba0ebff9c50bbc [id=8a0a0d875f5e4f5fbcbb74938698fbe3]) successfully deleted
I20260317 21:19:03.691105 7866 tablet_replica.cc:333] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:03.693553 7869 raft_consensus.cc:2243] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:03.694111 7869 raft_consensus.cc:2272] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:03.700740 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 7fc38947219047b5abdf71af5554a987 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fc38947219047b5abdf71af5554a987 already in progress: creating tablet
I20260317 21:19:03.701982 7866 raft_consensus.cc:2243] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:03.702704 7866 raft_consensus.cc:2272] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.704317 7869 ts_tablet_manager.cc:1916] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:03.717336 7459 tablet_bootstrap.cc:492] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:03.717811 7459 ts_tablet_manager.cc:1403] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.457s user 0.012s sys 0.000s
I20260317 21:19:03.722669 7459 raft_consensus.cc:359] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.723439 7459 raft_consensus.cc:385] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.723693 7459 raft_consensus.cc:740] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.726032 7459 consensus_queue.cc:260] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.731015 7467 raft_consensus.cc:359] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.731737 7467 raft_consensus.cc:385] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.732069 7467 raft_consensus.cc:740] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.733716 7459 ts_tablet_manager.cc:1434] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.016s user 0.006s sys 0.001s
I20260317 21:19:03.738571 7462 tablet_bootstrap.cc:492] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:03.743990 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 82f8dd260a134810a50465e4a77848ca (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]) successfully deleted
I20260317 21:19:03.683126 7791 ts_tablet_manager.cc:1929] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:03.744961 7791 log.cc:1199] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/163cc8c04ca94a269c7ae2efaa162525
I20260317 21:19:03.745812 7462 ts_tablet_manager.cc:1403] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.240s user 0.010s sys 0.019s
I20260317 21:19:03.757269 7791 ts_tablet_manager.cc:1950] T 163cc8c04ca94a269c7ae2efaa162525 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:03.763332 7161 tablet_service.cc:1558] Processing DeleteTablet for tablet c9b7f7c864004b9abde0a0201e196165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:03.777609 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_17cf4ddfbd3d47ba99971456fc3873f6" } modify_external_catalogs: true
I20260317 21:19:03.777693 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc929bae186841d2aba674659e457301" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:03.778173 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_17cf4ddfbd3d47ba99971456fc3873f6" } modify_external_catalogs: true
W20260317 21:19:03.779168 7120 leader_election.cc:343] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.779992 7120 leader_election.cc:304] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.780956 7937 raft_consensus.cc:2749] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.791887 7467 consensus_queue.cc:260] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.793161 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 935faf7f9129440a91635bfa96191cd6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:56 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:03.793071 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f666788fa2764510b242c535c4f50597"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:03.795030 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f666788fa2764510b242c535c4f50597 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:03.796684 7467 ts_tablet_manager.cc:1434] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.117s user 0.007s sys 0.000s
I20260317 21:19:03.797317 7878 raft_consensus.cc:493] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.797837 7878 raft_consensus.cc:515] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.804338 7878 leader_election.cc:290] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:03.805652 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:03.809095 7264 leader_election.cc:343] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.811961 7459 tablet_bootstrap.cc:492] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:03.812988 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:03.817389 7467 tablet_bootstrap.cc:492] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:03.818660 6669 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:03.819038 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 139) took 3963 ms. Trace:
I20260317 21:19:03.821856 7459 tablet_bootstrap.cc:654] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:03.822263 7261 leader_election.cc:343] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.822700 7261 leader_election.cc:304] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.819375 7791 rpcz_store.cc:276] 0317 21:18:59.855548 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.868087 (+ 12539us) service_pool.cc:225] Handling call
0317 21:19:03.359484 (+3491397us) ts_tablet_manager.cc:1151] Deleting tablet 163cc8c04ca94a269c7ae2efaa162525
0317 21:19:03.359537 (+ 53us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.630159 (+270622us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.682733 (+ 52574us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:03.782165 (+ 99432us) spinlock_profiling.cc:243] Waited 10.8 ms on lock 0x7b5800000098. stack: 00007f15f787f423 00007f15f787f320 00007f15f73f4574 0000000000509544 00000000005094d9 00007f15f76f1911 00007f15f7873e61 00007f15f7874492 00007f16006db219 00007f15ff92b1d8 00007f15ff9f29f2 00007f15ffa23510 00007f15ffa09add 00007f15ffa09a89 00007f15ffa09a19 00007f15ffa099e1
0317 21:19:03.818979 (+ 36814us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":24946944,"tablet-delete.queue_time_us":3398357}
I20260317 21:19:03.842489 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 163cc8c04ca94a269c7ae2efaa162525 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]) successfully deleted
I20260317 21:19:03.843683 7791 tablet_replica.cc:333] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:03.844518 7791 raft_consensus.cc:2243] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:03.846040 7791 pending_rounds.cc:70] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:03.859714 7467 tablet_bootstrap.cc:654] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:03.862754 6671 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 788f0ee1f7de47a49c5601c9b1494885 consensus state has no leader
I20260317 21:19:03.874603 7957 raft_consensus.cc:2749] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.875206 7957 raft_consensus.cc:493] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:03.875617 7957 raft_consensus.cc:515] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.877568 7957 leader_election.cc:290] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:03.878943 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:03.880486 7261 leader_election.cc:343] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.885386 7462 raft_consensus.cc:359] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.886360 7462 raft_consensus.cc:385] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.886854 7462 raft_consensus.cc:740] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.879206 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:03.898660 7467 tablet_bootstrap.cc:492] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:03.899134 7467 ts_tablet_manager.cc:1403] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.082s user 0.009s sys 0.000s
W20260317 21:19:03.901678 7264 leader_election.cc:343] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:03.902144 7264 leader_election.cc:304] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:03.904060 7878 raft_consensus.cc:2749] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:03.904764 7791 pending_rounds.cc:77] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412463005515776 op_type: NO_OP noop_request { }
I20260317 21:19:03.905261 7791 raft_consensus.cc:2889] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
W20260317 21:19:03.905927 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 123909us
I20260317 21:19:03.905819 7791 raft_consensus.cc:2272] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:03.908304 7791 ts_tablet_manager.cc:1916] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:03.911119 7467 raft_consensus.cc:359] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.911801 7467 raft_consensus.cc:385] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.912066 7467 raft_consensus.cc:740] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.922821 7467 consensus_queue.cc:260] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.926584 7866 ts_tablet_manager.cc:1916] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:03.930114 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8d94620882364ed3929ee314814910c9 consensus state has no leader
I20260317 21:19:03.930977 7467 ts_tablet_manager.cc:1434] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
I20260317 21:19:03.910535 7459 tablet_bootstrap.cc:492] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:03.939424 7467 tablet_bootstrap.cc:492] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:19:03.943739 6662 consensus_peers.cc:597] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 -> Peer 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857): Couldn't send request to peer 47e1c983d0224428a9a7b147a0e5e4ef. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:38857: connect: Connection refused (error 111). This is attempt 26: this message will repeat every 5th retry.
I20260317 21:19:03.944321 7459 ts_tablet_manager.cc:1403] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.133s user 0.011s sys 0.000s
I20260317 21:19:03.947212 7459 raft_consensus.cc:359] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.948132 7459 raft_consensus.cc:385] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:03.948397 7459 raft_consensus.cc:740] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:03.949474 7459 consensus_queue.cc:260] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:03.967025 7467 tablet_bootstrap.cc:654] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:03.982081 7459 ts_tablet_manager.cc:1434] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.037s user 0.003s sys 0.003s
I20260317 21:19:03.986168 7459 tablet_bootstrap.cc:492] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:03.995805 7467 tablet_bootstrap.cc:492] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:04.000989 7467 ts_tablet_manager.cc:1403] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.004s
I20260317 21:19:04.002606 7976 tablet.cc:1723] T b5a797b22cf34483b0061955da6b2634 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:04.004477 7462 consensus_queue.cc:260] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.016988 7467 raft_consensus.cc:359] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.020133 7467 raft_consensus.cc:385] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.020735 7467 raft_consensus.cc:740] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:04.026829 7467 consensus_queue.cc:260] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.030278 7467 ts_tablet_manager.cc:1434] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.017s user 0.005s sys 0.000s
I20260317 21:19:04.034572 6674 master_service.cc:954] ReplaceTablet: received request to replace tablet eb332f981fc84a178fadae2f4908bdde from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:04.040256 7467 tablet_bootstrap.cc:492] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:04.052039 7162 tablet_service.cc:1511] Processing CreateTablet for tablet dfff4c8478b64436a666a14c133c36d3 (DEFAULT_TABLE table=default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.053120 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfff4c8478b64436a666a14c133c36d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.057040 6671 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:04.068601 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 70472ad137f34237a5a67f74c94aa36b (DEFAULT_TABLE table=default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.069111 6671 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_638c2b3200334f309baef743312b646c" } new_table_name: "default.Table_9a3ec369ff814877a0ff250609d2803e" modify_external_catalogs: true
I20260317 21:19:04.069633 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70472ad137f34237a5a67f74c94aa36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.078228 7305 tablet_service.cc:1511] Processing CreateTablet for tablet dfff4c8478b64436a666a14c133c36d3 (DEFAULT_TABLE table=default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.081245 7467 tablet_bootstrap.cc:654] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.086395 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 70472ad137f34237a5a67f74c94aa36b (DEFAULT_TABLE table=default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.090444 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 4737071c2aec4449a05bead6af7ad32b (DEFAULT_TABLE table=default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.091496 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4737071c2aec4449a05bead6af7ad32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.092232 6670 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]). Expected version 1 got 0
W20260317 21:19:04.025116 6971 consensus_peers.cc:597] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82f8dd260a134810a50465e4a77848ca. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:04.100580 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 75589.1us
I20260317 21:19:04.049247 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet d2f7c1058f8245428f38fe3245621b75 with delete_type TABLET_DATA_DELETED (replaced by tablet e22772281bd74754bb969e87a70d8077) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.060977 7015 tablet_service.cc:1511] Processing CreateTablet for tablet dfff4c8478b64436a666a14c133c36d3 (DEFAULT_TABLE table=default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.102414 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfff4c8478b64436a666a14c133c36d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.108886 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 16f64073188f4d06859e1a701df73242
I20260317 21:19:04.082888 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 70472ad137f34237a5a67f74c94aa36b (DEFAULT_TABLE table=default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.111294 7866 ts_tablet_manager.cc:1929] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260317 21:19:04.111370 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f64073188f4d06859e1a701df73242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.111760 7866 log.cc:1199] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/2562cc2a3bb2430981c88475f9e1daa1
I20260317 21:19:04.111848 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70472ad137f34237a5a67f74c94aa36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.121508 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f64073188f4d06859e1a701df73242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.123255 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 16f64073188f4d06859e1a701df73242 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:04.124786 7943 raft_consensus.cc:493] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.125254 7943 raft_consensus.cc:515] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.127089 7943 leader_election.cc:290] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:04.129629 6971 leader_election.cc:343] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:19:04.087936 7459 tablet_bootstrap.cc:654] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.091858 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 4737071c2aec4449a05bead6af7ad32b (DEFAULT_TABLE table=default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.137409 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4737071c2aec4449a05bead6af7ad32b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:04.140177 6973 consensus_peers.cc:597] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 82f8dd260a134810a50465e4a77848ca. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:04.140758 6973 leader_election.cc:343] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: b6fb7cb082cf47ce8cf2a98146633af7
I20260317 21:19:04.141084 6973 leader_election.cc:304] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:04.141122 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_9a942b013ad0470bb7e5ff4149ca0498"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:04.141319 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1bd5ef7ffcac4314b52b8df4265dfb67"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:04.142628 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a942b013ad0470bb7e5ff4149ca0498 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:04.142959 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:04.143774 7943 raft_consensus.cc:2749] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:04.147188 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_a571e2a5fe2e4b1c92f568eb955c8a38" } new_table_name: "default.Table_4e0de322dbf141f38753722166148d81" modify_external_catalogs: true
I20260317 21:19:04.153517 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70472ad137f34237a5a67f74c94aa36b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.154354 7878 raft_consensus.cc:493] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.154980 7878 raft_consensus.cc:515] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.112499 7866 ts_tablet_manager.cc:1950] T 2562cc2a3bb2430981c88475f9e1daa1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:04.161864 7878 leader_election.cc:290] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:04.164075 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:04.164858 7181 raft_consensus.cc:2468] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:04.166311 7261 leader_election.cc:304] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:04.168521 7878 raft_consensus.cc:2804] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:04.168959 7878 raft_consensus.cc:493] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:04.169291 7878 raft_consensus.cc:3060] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:04.173262 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:04.174571 7264 leader_election.cc:343] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:04.177372 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dfff4c8478b64436a666a14c133c36d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.182238 7866 tablet_replica.cc:333] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:04.182416 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]) successfully deleted
I20260317 21:19:04.183246 7866 raft_consensus.cc:2243] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.184206 7866 raft_consensus.cc:2272] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.186896 7791 ts_tablet_manager.cc:1929] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:04.187307 7791 log.cc:1199] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/60f3d32e1f7649fa92861f51134cf000
I20260317 21:19:04.188058 7791 ts_tablet_manager.cc:1950] T 60f3d32e1f7649fa92861f51134cf000 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:04.194716 7459 tablet_bootstrap.cc:492] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:04.195148 7459 ts_tablet_manager.cc:1403] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.209s user 0.005s sys 0.005s
I20260317 21:19:04.197330 7459 raft_consensus.cc:359] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.197964 7459 raft_consensus.cc:385] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.198200 7459 raft_consensus.cc:740] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:04.198733 7459 consensus_queue.cc:260] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.200663 7459 ts_tablet_manager.cc:1434] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.005s user 0.005s sys 0.000s
I20260317 21:19:04.204377 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.211817 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 140) took 4355 ms. Trace:
I20260317 21:19:04.212159 7791 rpcz_store.cc:276] 0317 21:18:59.855945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:59.931245 (+ 75300us) service_pool.cc:225] Handling call
0317 21:19:03.843517 (+3912272us) ts_tablet_manager.cc:1151] Deleting tablet 60f3d32e1f7649fa92861f51134cf000
0317 21:19:03.843564 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.000119 (+156555us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.186644 (+186525us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.211738 (+ 25094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3818128}
I20260317 21:19:04.215258 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 60f3d32e1f7649fa92861f51134cf000 (table default.Table_7a69602e996b4ca6aa936f73b9b24c34 [id=7680260305df414a9e15cd0f1f47a838]) successfully deleted
I20260317 21:19:04.214671 7866 ts_tablet_manager.cc:1916] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.224977 7878 raft_consensus.cc:515] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.240075 7943 raft_consensus.cc:493] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.240705 7943 raft_consensus.cc:515] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.243155 7943 leader_election.cc:290] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:04.250981 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3c680f76c8ca4c0298b9f1e3487081f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:04.253638 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c680f76c8ca4c0298b9f1e3487081f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:04.258267 6973 leader_election.cc:343] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 589aadc7897b422e90d3f7caed76ab59
W20260317 21:19:04.258733 6971 leader_election.cc:343] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 589aadc7897b422e90d3f7caed76ab59
I20260317 21:19:04.259327 6971 leader_election.cc:304] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:04.260396 7943 raft_consensus.cc:2749] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:04.261770 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e86269e06d1d41d6bcc0379135d9ccbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:04.263957 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e86269e06d1d41d6bcc0379135d9ccbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:04.276937 7410 log.cc:927] Time spent T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Append to log took a long time: real 0.093s user 0.000s sys 0.001s
I20260317 21:19:04.297448 7791 tablet_replica.cc:333] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:04.297696 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet dc929bae186841d2aba674659e457301 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.298162 7791 raft_consensus.cc:2243] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.298686 7791 raft_consensus.cc:2272] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.300714 7791 ts_tablet_manager.cc:1916] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.302825 7459 tablet_bootstrap.cc:492] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:04.308770 7459 tablet_bootstrap.cc:654] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.319557 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 4737071c2aec4449a05bead6af7ad32b (DEFAULT_TABLE table=default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.332691 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4737071c2aec4449a05bead6af7ad32b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.337747 6670 master_service.cc:954] ReplaceTablet: received request to replace tablet 2607b9978c32466eb6dc8b8acfb429ce from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:04.340054 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
W20260317 21:19:04.341614 7264 leader_election.cc:343] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:04.341967 7462 ts_tablet_manager.cc:1434] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.595s user 0.011s sys 0.000s
W20260317 21:19:04.342238 7462 ts_tablet_manager.cc:1467] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Tablet startup took 596ms
W20260317 21:19:04.342496 7462 ts_tablet_manager.cc:1469] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Trace:
0317 21:18:58.911008 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:18:58.911139 (+ 131us) service_pool.cc:225] Handling call
0317 21:18:58.912312 (+ 1173us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 21:18:58.912381 (+ 69us) ts_tablet_manager.cc:682] Creating new metadata...
0317 21:18:58.920423 (+ 8042us) tablet_metadata.cc:641] Metadata flushed
0317 21:18:58.996480 (+ 76057us) tablet_replica.cc:206] Creating consensus instance
0317 21:18:58.996867 (+ 387us) inbound_call.cc:173] Queueing success response
0317 21:19:03.505288 (+4508421us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 21:19:03.747075 (+241787us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 21:19:03.805344 (+ 58269us) tablet_replica.cc:261] Starting instrumentation
0317 21:19:03.805829 (+ 485us) tablet_replica.cc:280] Starting consensus
0317 21:19:04.229687 (+423858us) spinlock_profiling.cc:243] Waited 38.6 ms on lock 0x7b54000d3328. stack: 00007f8cd9137423 00007f8cd9137320 00007f8cd8cac574 0000000000509544 00000000005094d9 00007f8ce05ca951 00007f8ce0652e43 00007f8ce12a902a 00007f8ce12bcfa1 00007f8ce12bcf49 00007f8ce12bced9 00007f8ce12bcea1 00007f8ce12bc17d 00000000005ab084 00000000005aaeb9 00007f8cd9169405
Metrics: {"raft.queue_time_us":161,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":203,"spinlock_wait_cycles":88725248,"tablet-open.queue_time_us":4508470}
I20260317 21:19:04.357161 7467 tablet_bootstrap.cc:492] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:04.357569 7467 ts_tablet_manager.cc:1403] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.318s user 0.042s sys 0.042s
I20260317 21:19:04.362202 7162 tablet_service.cc:1511] Processing CreateTablet for tablet ae1b2a549af6485ca3faee3f0ad845a2 (DEFAULT_TABLE table=default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.363240 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1b2a549af6485ca3faee3f0ad845a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.364063 7008 tablet_service.cc:1511] Processing CreateTablet for tablet ae1b2a549af6485ca3faee3f0ad845a2 (DEFAULT_TABLE table=default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.365239 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1b2a549af6485ca3faee3f0ad845a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.373852 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:04.374634 7181 raft_consensus.cc:3060] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:04.378861 7459 tablet_bootstrap.cc:492] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:04.378398 7467 raft_consensus.cc:359] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.379278 7459 ts_tablet_manager.cc:1403] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.077s user 0.011s sys 0.000s
I20260317 21:19:04.379465 7467 raft_consensus.cc:385] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.379887 7467 raft_consensus.cc:740] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:04.380756 7467 consensus_queue.cc:260] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.382331 7459 raft_consensus.cc:359] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.382957 7459 raft_consensus.cc:385] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.383225 7459 raft_consensus.cc:740] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:04.383922 7459 consensus_queue.cc:260] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.387275 7181 raft_consensus.cc:2468] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:04.390976 7878 leader_election.cc:290] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:04.395255 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 1c3beb9232d74e45ab713ae8241eec4c
I20260317 21:19:04.400038 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.403263 7791 ts_tablet_manager.cc:1929] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:04.412799 7791 log.cc:1199] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/478f55d4c3034be0b1ef336a173c55b2
I20260317 21:19:04.421113 7791 ts_tablet_manager.cc:1950] T 478f55d4c3034be0b1ef336a173c55b2 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:04.422745 7866 ts_tablet_manager.cc:1929] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:04.424983 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 146) took 3999 ms. Trace:
W20260317 21:19:04.431138 6671 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7eb432b6bccf4e32a2fa9d4979586c52 consensus state has no leader
I20260317 21:19:04.433733 7866 log.cc:1199] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/59fce28afbeb4d8eadf63f9a97d1183d
I20260317 21:19:04.431409 7303 tablet_service.cc:1511] Processing CreateTablet for tablet ae1b2a549af6485ca3faee3f0ad845a2 (DEFAULT_TABLE table=default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.434348 7261 leader_election.cc:304] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:04.435070 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1b2a549af6485ca3faee3f0ad845a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.435140 6675 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:04.435528 7878 raft_consensus.cc:2804] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
W20260317 21:19:04.439620 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 829bc3959a9e4c2dbb6277cde5969b0e consensus state has no leader
I20260317 21:19:04.390601 7869 ts_tablet_manager.cc:1929] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:04.397212 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:04.441372 7869 log.cc:1199] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/935faf7f9129440a91635bfa96191cd6
I20260317 21:19:04.442552 7869 ts_tablet_manager.cc:1950] T 935faf7f9129440a91635bfa96191cd6 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:04.425308 7462 tablet_bootstrap.cc:492] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:04.448199 6674 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet eb332f981fc84a178fadae2f4908bdde
I20260317 21:19:04.449000 7866 ts_tablet_manager.cc:1950] T 59fce28afbeb4d8eadf63f9a97d1183d P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:04.449334 7791 rpcz_store.cc:276] 0317 21:19:00.425372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:00.425486 (+ 114us) service_pool.cc:225] Handling call
0317 21:19:04.213068 (+3787582us) ts_tablet_manager.cc:1151] Deleting tablet 478f55d4c3034be0b1ef336a173c55b2
0317 21:19:04.297312 (+ 84244us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.331115 (+ 33803us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.402966 (+ 71851us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.424924 (+ 21958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3786977}
I20260317 21:19:04.450244 6674 catalog_manager.cc:6741] ReplaceTablet: tablet eb332f981fc84a178fadae2f4908bdde deleted and replaced by tablet f6cd3f64f9b94d089b8ecf2462007ee6
I20260317 21:19:04.450786 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet eb332f981fc84a178fadae2f4908bdde with delete_type TABLET_DATA_DELETED (replaced by tablet f6cd3f64f9b94d089b8ecf2462007ee6) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:04.406339 7459 ts_tablet_manager.cc:1434] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260317 21:19:04.452222 7459 tablet_bootstrap.cc:492] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:04.458869 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 4af622a9190148ee80b2d5d3d7a92a20 (DEFAULT_TABLE table=default.Table_0d5bd4305663413496306f96a3b22926 [id=6f8ac04310bf451292f89b1fb3ad6489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.459337 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bc634ee1d0df4afcb027ea787dce81b5" } modify_external_catalogs: true
I20260317 21:19:04.459969 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bc634ee1d0df4afcb027ea787dce81b5" } modify_external_catalogs: true
I20260317 21:19:04.460156 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af622a9190148ee80b2d5d3d7a92a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.461081 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 149) took 3801 ms. Trace:
I20260317 21:19:04.461119 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 478f55d4c3034be0b1ef336a173c55b2 (table default.Table_76a974b216ba482e9783ae7f126c5d31 [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]) successfully deleted
I20260317 21:19:04.461395 7791 rpcz_store.cc:276] 0317 21:19:00.659657 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:00.659781 (+ 124us) service_pool.cc:225] Handling call
0317 21:19:04.460728 (+3800947us) ts_tablet_manager.cc:1151] Deleting tablet 0eee8b5588c24661889c593797c94269
0317 21:19:04.460774 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.461028 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3800273}
I20260317 21:19:04.462685 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 151) took 3706 ms. Trace:
I20260317 21:19:04.462972 7791 rpcz_store.cc:276] 0317 21:19:00.756825 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:00.756971 (+ 146us) service_pool.cc:225] Handling call
0317 21:19:04.462188 (+3705217us) ts_tablet_manager.cc:1151] Deleting tablet 3e2f891e712341868d94bab1dd72a9e1
0317 21:19:04.462227 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.462633 (+ 406us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3617293}
W20260317 21:19:04.463349 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 0eee8b5588c24661889c593797c94269 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0eee8b5588c24661889c593797c94269 already in progress: creating tablet
I20260317 21:19:04.464118 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 153) took 3662 ms. Trace:
W20260317 21:19:04.464445 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 3e2f891e712341868d94bab1dd72a9e1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3e2f891e712341868d94bab1dd72a9e1 already in progress: creating tablet
I20260317 21:19:04.444869 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 193) took 1943 ms. Trace:
I20260317 21:19:04.464972 7869 rpcz_store.cc:276] 0317 21:19:02.501724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.501880 (+ 156us) service_pool.cc:225] Handling call
0317 21:19:03.690706 (+1188826us) ts_tablet_manager.cc:1151] Deleting tablet 935faf7f9129440a91635bfa96191cd6
0317 21:19:03.690770 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:03.776886 (+ 86116us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.390342 (+613456us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.444803 (+ 54461us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1005571}
I20260317 21:19:04.466290 7866 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 204) took 1037 ms. Trace:
I20260317 21:19:04.466713 7866 rpcz_store.cc:276] 0317 21:19:03.432998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.433338 (+ 340us) service_pool.cc:225] Handling call
0317 21:19:04.182072 (+748734us) ts_tablet_manager.cc:1151] Deleting tablet 59fce28afbeb4d8eadf63f9a97d1183d
0317 21:19:04.182119 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.343125 (+161006us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.422498 (+ 79373us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.466171 (+ 43673us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":748135}
I20260317 21:19:04.468817 7866 tablet_replica.cc:333] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:04.471145 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 2607b9978c32466eb6dc8b8acfb429ce
I20260317 21:19:04.474574 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 2607b9978c32466eb6dc8b8acfb429ce with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1819f74297406e8ebb28ae847abdd3) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.477873 6670 catalog_manager.cc:6741] ReplaceTablet: tablet 2607b9978c32466eb6dc8b8acfb429ce deleted and replaced by tablet 9a1819f74297406e8ebb28ae847abdd3
I20260317 21:19:04.477564 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 59fce28afbeb4d8eadf63f9a97d1183d (table default.Table_6d4e07837b8945f981be0ae07f5b4ea0 [id=2e0f6bf62c664a6cb3f2bdad9a74e11b]) successfully deleted
I20260317 21:19:04.473337 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 2607b9978c32466eb6dc8b8acfb429ce with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1819f74297406e8ebb28ae847abdd3) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:04.479691 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet c9b7f7c864004b9abde0a0201e196165 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7f7c864004b9abde0a0201e196165 already in progress: creating tablet
W20260317 21:19:04.480418 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 16f64073188f4d06859e1a701df73242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f64073188f4d06859e1a701df73242 already in progress: creating tablet
I20260317 21:19:04.481442 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 935faf7f9129440a91635bfa96191cd6 (table default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]) successfully deleted
I20260317 21:19:04.485318 7459 tablet_bootstrap.cc:654] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.485896 7869 tablet_replica.cc:333] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:04.486728 7869 raft_consensus.cc:2243] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.487287 7869 raft_consensus.cc:2272] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.492095 7990 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.124:38857 every 8 connection(s)
I20260317 21:19:04.494932 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 4af622a9190148ee80b2d5d3d7a92a20 (DEFAULT_TABLE table=default.Table_0d5bd4305663413496306f96a3b22926 [id=6f8ac04310bf451292f89b1fb3ad6489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.473023 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet eb332f981fc84a178fadae2f4908bdde with delete_type TABLET_DATA_DELETED (replaced by tablet f6cd3f64f9b94d089b8ecf2462007ee6) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.475016 7462 tablet_bootstrap.cc:654] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.448890 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.474377 7878 raft_consensus.cc:697] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:04.464398 7791 rpcz_store.cc:276] 0317 21:19:00.802148 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:00.802301 (+ 153us) service_pool.cc:225] Handling call
0317 21:19:04.463757 (+3661456us) ts_tablet_manager.cc:1151] Deleting tablet 35023195b9094d58827921a1dcb2ff05
0317 21:19:04.463798 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.464059 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3606733}
I20260317 21:19:04.504493 7878 consensus_queue.cc:237] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:04.506446 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 35023195b9094d58827921a1dcb2ff05 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 35023195b9094d58827921a1dcb2ff05 already in progress: creating tablet
I20260317 21:19:04.505306 7866 raft_consensus.cc:2243] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.508339 7866 raft_consensus.cc:2272] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.500429 7443 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.124:38857
I20260317 21:19:04.511937 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet eb332f981fc84a178fadae2f4908bdde with delete_type TABLET_DATA_DELETED (replaced by tablet f6cd3f64f9b94d089b8ecf2462007ee6) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.521740 7943 raft_consensus.cc:493] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.522383 7943 raft_consensus.cc:515] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.527540 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 6ab5f3dc7836425d917bd09f1e6bfe90
I20260317 21:19:04.531970 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:04 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.533146 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af622a9190148ee80b2d5d3d7a92a20. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:04.537962 6973 leader_election.cc:343] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: dc929bae186841d2aba674659e457301
I20260317 21:19:04.539347 7943 leader_election.cc:290] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:04.541056 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 175) took 3023 ms. Trace:
I20260317 21:19:04.541471 7791 rpcz_store.cc:276] 0317 21:19:01.517416 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:01.518043 (+ 627us) service_pool.cc:225] Handling call
0317 21:19:04.532567 (+3014524us) ts_tablet_manager.cc:1151] Deleting tablet 06877bd7155842b9a5ae4a23ea68ada0
0317 21:19:04.540804 (+ 8237us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.541003 (+ 199us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2988842}
W20260317 21:19:04.542863 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 06877bd7155842b9a5ae4a23ea68ada0 because the tablet was not found. No further retry: Not found: Tablet not found: 06877bd7155842b9a5ae4a23ea68ada0
I20260317 21:19:04.544013 7791 tablet_replica.cc:333] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:04.490948 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 2607b9978c32466eb6dc8b8acfb429ce with delete_type TABLET_DATA_DELETED (replaced by tablet 9a1819f74297406e8ebb28ae847abdd3) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.520790 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 4af622a9190148ee80b2d5d3d7a92a20 (DEFAULT_TABLE table=default.Table_0d5bd4305663413496306f96a3b22926 [id=6f8ac04310bf451292f89b1fb3ad6489]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.531279 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:04 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.556041 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4af622a9190148ee80b2d5d3d7a92a20. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.555955 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:04.546118 7010 tablet_service.cc:1511] Processing CreateTablet for tablet f69403b3b811419e87916a381656e1ec (DEFAULT_TABLE table=default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.562143 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69403b3b811419e87916a381656e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.571655 7869 ts_tablet_manager.cc:1916] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.544837 7791 raft_consensus.cc:2243] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.573166 7791 raft_consensus.cc:2272] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.575806 7791 ts_tablet_manager.cc:1916] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.577227 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc929bae186841d2aba674659e457301" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:04.573256 7459 tablet_bootstrap.cc:492] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:04.577844 7181 raft_consensus.cc:2468] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:04.578248 7459 ts_tablet_manager.cc:1403] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.126s user 0.008s sys 0.003s
I20260317 21:19:04.579134 6971 leader_election.cc:304] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:04.580157 7943 raft_consensus.cc:493] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.580924 7443 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
I20260317 21:19:04.582849 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 7443
I20260317 21:19:04.584595 7866 ts_tablet_manager.cc:1916] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.580760 7943 raft_consensus.cc:515] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.591849 7459 raft_consensus.cc:359] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.610438 7459 raft_consensus.cc:385] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.610771 7459 raft_consensus.cc:740] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:04.592365 6973 leader_election.cc:343] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 935faf7f9129440a91635bfa96191cd6
I20260317 21:19:04.611446 7459 consensus_queue.cc:260] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.607177 7943 leader_election.cc:290] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:04.613456 6971 leader_election.cc:343] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Not found: Tablet not found: 935faf7f9129440a91635bfa96191cd6
I20260317 21:19:04.615065 6971 leader_election.cc:304] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:04.616191 8000 raft_consensus.cc:2749] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:04.610020 7994 raft_consensus.cc:2804] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:04.617242 7994 raft_consensus.cc:493] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:04.617677 7994 raft_consensus.cc:3060] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:04.630659 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 2a221dfad830428db2329d6e543f41dc (table default.Table_5ebd4f0e4c284b4caac2e5f0cddf4e07 [id=2d22abe029d348c484e6aff5df8b3d29], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 2a221dfad830428db2329d6e543f41dc consensus state has no leader
I20260317 21:19:04.632903 7467 ts_tablet_manager.cc:1434] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.275s user 0.009s sys 0.004s
I20260317 21:19:04.634795 7459 ts_tablet_manager.cc:1434] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.056s user 0.004s sys 0.001s
I20260317 21:19:04.636041 7459 tablet_bootstrap.cc:492] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:04.635984 7866 ts_tablet_manager.cc:1929] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:04.636351 7866 log.cc:1199] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:04.636255 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_411f7f4c71494d26b681f5e76fa36214" } modify_external_catalogs: true
I20260317 21:19:04.637101 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_411f7f4c71494d26b681f5e76fa36214" } modify_external_catalogs: true
I20260317 21:19:04.642784 7866 ts_tablet_manager.cc:1950] T 3e2f891e712341868d94bab1dd72a9e1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:04.649174 7866 tablet_replica.cc:333] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:04.650298 7866 raft_consensus.cc:2243] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.652951 7866 raft_consensus.cc:2272] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.655833 7298 tablet_service.cc:1511] Processing CreateTablet for tablet f69403b3b811419e87916a381656e1ec (DEFAULT_TABLE table=default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.662207 7462 tablet_bootstrap.cc:492] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:04.662791 7462 ts_tablet_manager.cc:1403] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.238s user 0.009s sys 0.002s
I20260317 21:19:04.665504 7462 raft_consensus.cc:359] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.671841 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e86269e06d1d41d6bcc0379135d9ccbe" } new_table_name: "default.Table_8ec60a799e6c4582bf9873c1762dca5c" modify_external_catalogs: true
I20260317 21:19:04.675179 7994 raft_consensus.cc:515] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.687809 7467 tablet_bootstrap.cc:492] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:04.666919 7866 ts_tablet_manager.cc:1916] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.688977 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69403b3b811419e87916a381656e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.701838 6670 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f666788fa2764510b242c535c4f50597" } new_table_name: "default.Table_0d594c7cc868438fa7b25513ce01f0e9" modify_external_catalogs: true
I20260317 21:19:04.694996 7791 ts_tablet_manager.cc:1929] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:04.708850 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dc929bae186841d2aba674659e457301" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:04.709429 7181 raft_consensus.cc:3060] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:04.710971 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4ba2b776e70f400bb96eedf80ddfc212"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:04.712270 7462 raft_consensus.cc:385] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.712569 7462 raft_consensus.cc:740] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:04.706833 7791 log.cc:1199] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/b6fb7cb082cf47ce8cf2a98146633af7
W20260317 21:19:04.712579 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ba2b776e70f400bb96eedf80ddfc212 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:04.713410 7791 ts_tablet_manager.cc:1950] T b6fb7cb082cf47ce8cf2a98146633af7 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:04.713238 7462 consensus_queue.cc:260] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.713541 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4c8c24e29f304d99b8a6540fc42b887a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:04.714221 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 176) took 3196 ms. Trace:
W20260317 21:19:04.715454 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4c8c24e29f304d99b8a6540fc42b887a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:04.720468 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3ab39e76bb8c42b8b4917eaf4ab5a2d9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:04.714546 7791 rpcz_store.cc:276] 0317 21:19:01.517841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:01.518897 (+ 1056us) service_pool.cc:225] Handling call
0317 21:19:04.543854 (+3024957us) ts_tablet_manager.cc:1151] Deleting tablet b6fb7cb082cf47ce8cf2a98146633af7
0317 21:19:04.543896 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.642828 (+ 98932us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.694758 (+ 51930us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.714159 (+ 19401us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3006262}
I20260317 21:19:04.721552 7791 tablet_replica.cc:333] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:04.722072 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ab39e76bb8c42b8b4917eaf4ab5a2d9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:04.722245 7791 raft_consensus.cc:2243] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.722690 7791 raft_consensus.cc:2272] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.722627 7937 raft_consensus.cc:493] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.723100 7937 raft_consensus.cc:515] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.723836 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_7112fbaf5a99486b95479c3f8f64a8cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:04.724414 7791 ts_tablet_manager.cc:1916] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:04.725417 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7112fbaf5a99486b95479c3f8f64a8cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:04.726879 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:04.727505 7326 raft_consensus.cc:2468] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:04.728590 7937 leader_election.cc:290] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:04.730515 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:04 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:04.732100 7162 tablet_service.cc:1511] Processing CreateTablet for tablet f69403b3b811419e87916a381656e1ec (DEFAULT_TABLE table=default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.733165 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f69403b3b811419e87916a381656e1ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.736953 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:04.737112 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 4495961a559549629dedf8ce032abc9b (DEFAULT_TABLE table=default.Table_67179be887be4140976c9d859cba3a88 [id=539b531cf22a4816b0aff0f3908b0439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.738075 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4495961a559549629dedf8ce032abc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.739346 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:04.739982 7038 raft_consensus.cc:2468] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:04.743494 7181 raft_consensus.cc:2468] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:04.744208 7120 leader_election.cc:304] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:04.744679 6971 leader_election.cc:304] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:04.744916 7937 raft_consensus.cc:2804] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:04.745440 7999 raft_consensus.cc:2804] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:04.746140 7999 raft_consensus.cc:697] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:04.745234 7937 raft_consensus.cc:493] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:04.746821 7937 raft_consensus.cc:3060] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:04.747447 7999 consensus_queue.cc:237] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.753752 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet b6fb7cb082cf47ce8cf2a98146633af7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]) successfully deleted
I20260317 21:19:04.756484 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 4495961a559549629dedf8ce032abc9b (DEFAULT_TABLE table=default.Table_67179be887be4140976c9d859cba3a88 [id=539b531cf22a4816b0aff0f3908b0439]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:04.759651 6973 leader_election.cc:343] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: dc929bae186841d2aba674659e457301
I20260317 21:19:04.705850 7459 tablet_bootstrap.cc:654] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.761736 7994 leader_election.cc:290] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:04.768074 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4495961a559549629dedf8ce032abc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.782274 7462 ts_tablet_manager.cc:1434] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.119s user 0.005s sys 0.000s
I20260317 21:19:04.782862 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 3e2f891e712341868d94bab1dd72a9e1 (table default.Table_0c31d9e5b0bd4454acccb920047a5e73 [id=8fbd9827525947d58d75c279e4b92b6a]) successfully deleted
I20260317 21:19:04.785825 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6028353a5dad482db231700fb6461738" } modify_external_catalogs: true
I20260317 21:19:04.786393 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6028353a5dad482db231700fb6461738" } modify_external_catalogs: true
I20260317 21:19:04.788697 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet f749358db0f74ef78d9eee23b72bfe3e
I20260317 21:19:04.792331 7010 tablet_service.cc:1511] Processing CreateTablet for tablet d51e3d4702724484b776a95eb5acf7ef (DEFAULT_TABLE table=default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.793601 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51e3d4702724484b776a95eb5acf7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.793411 6666 catalog_manager.cc:5671] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67). New cstate: current_term: 1 leader_uuid: "2f539d1a4a584b8ca83928be49369bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } }
I20260317 21:19:04.798831 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 4495961a559549629dedf8ce032abc9b (DEFAULT_TABLE table=default.Table_67179be887be4140976c9d859cba3a88 [id=539b531cf22a4816b0aff0f3908b0439]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.800778 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4495961a559549629dedf8ce032abc9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.792308 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet f749358db0f74ef78d9eee23b72bfe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:04 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:04.801939 7459 tablet_bootstrap.cc:492] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:04.802348 7459 ts_tablet_manager.cc:1403] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.167s user 0.007s sys 0.005s
I20260317 21:19:04.808873 7462 tablet_bootstrap.cc:492] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:04.813915 7995 sys_catalog.cc:263] Verifying existing consensus state
I20260317 21:19:04.821930 7866 ts_tablet_manager.cc:1929] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:04.822382 7866 log.cc:1199] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/eb332f981fc84a178fadae2f4908bdde
I20260317 21:19:04.823083 7866 ts_tablet_manager.cc:1950] T eb332f981fc84a178fadae2f4908bdde P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:04.825477 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 (DEFAULT_TABLE table=default.Table_bc634ee1d0df4afcb027ea787dce81b5 [id=19d461771e63402d98c3803a6b7cbae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.826516 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab5f3dc7836425d917bd09f1e6bfe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.835489 7791 ts_tablet_manager.cc:1929] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:04.835862 7791 log.cc:1199] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/589aadc7897b422e90d3f7caed76ab59
W20260317 21:19:04.835903 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 50483.3us
I20260317 21:19:04.836508 7791 ts_tablet_manager.cc:1950] T 589aadc7897b422e90d3f7caed76ab59 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:04.837352 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 177) took 3294 ms. Trace:
I20260317 21:19:04.837658 7791 rpcz_store.cc:276] 0317 21:19:01.542713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:01.542852 (+ 139us) service_pool.cc:225] Handling call
0317 21:19:04.721403 (+3178551us) ts_tablet_manager.cc:1151] Deleting tablet 589aadc7897b422e90d3f7caed76ab59
0317 21:19:04.721444 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.793253 (+ 71809us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.835282 (+ 42029us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.837294 (+ 2012us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3177929}
I20260317 21:19:04.838794 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 589aadc7897b422e90d3f7caed76ab59 (table default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]) successfully deleted
I20260317 21:19:04.852578 7459 raft_consensus.cc:359] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.853281 7459 raft_consensus.cc:385] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:04.853587 7459 raft_consensus.cc:740] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:04.853438 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 178) took 3263 ms. Trace:
I20260317 21:19:04.855060 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet f749358db0f74ef78d9eee23b72bfe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:04 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:04.856619 7301 tablet_service.cc:1511] Processing CreateTablet for tablet d51e3d4702724484b776a95eb5acf7ef (DEFAULT_TABLE table=default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.857653 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51e3d4702724484b776a95eb5acf7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.859794 7462 tablet_bootstrap.cc:654] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.867547 7459 consensus_queue.cc:260] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.854225 7791 rpcz_store.cc:276] 0317 21:19:01.590612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:01.590808 (+ 196us) service_pool.cc:225] Handling call
0317 21:19:04.852902 (+3262094us) ts_tablet_manager.cc:1151] Deleting tablet 3d40549347f8463797d6791136883371
0317 21:19:04.852944 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.853375 (+ 431us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3261268}
I20260317 21:19:04.872893 7866 tablet_replica.cc:333] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
W20260317 21:19:04.873405 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 3d40549347f8463797d6791136883371 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3d40549347f8463797d6791136883371 already in progress: creating tablet
I20260317 21:19:04.873824 7866 raft_consensus.cc:2243] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:04.874307 7866 raft_consensus.cc:2272] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:04.875017 7791 tablet_replica.cc:333] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:04.875753 7791 raft_consensus.cc:2243] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:04.877143 7791 pending_rounds.cc:70] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:04.883479 7866 ts_tablet_manager.cc:1916] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.883929 7791 pending_rounds.cc:77] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412471237423104 op_type: NO_OP noop_request { }
I20260317 21:19:04.884596 7791 raft_consensus.cc:2889] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:04.885068 7791 raft_consensus.cc:2272] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:04.891270 6670 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e080308f93448138a4846ab6bec7dfa has no consensus state
W20260317 21:19:04.894801 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2152e5cbd2ce4159b0901ba2ffe41f40 has no consensus state
I20260317 21:19:04.899045 6671 master_service.cc:954] ReplaceTablet: received request to replace tablet 2b287c83b77548f19546927b05905746 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:04.894760 7791 ts_tablet_manager.cc:1916] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:04.894089 7994 raft_consensus.cc:493] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.908746 7994 raft_consensus.cc:515] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.910869 7994 leader_election.cc:290] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:04.886601 7459 ts_tablet_manager.cc:1434] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.069s user 0.005s sys 0.000s
W20260317 21:19:04.913044 6973 leader_election.cc:343] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: d2f7c1058f8245428f38fe3245621b75
I20260317 21:19:04.917862 7015 tablet_service.cc:1511] Processing CreateTablet for tablet eb3a61ed542741d88d89ef6b4c53e9e6 (DEFAULT_TABLE table=default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.918934 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3a61ed542741d88d89ef6b4c53e9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.929515 7459 tablet_bootstrap.cc:492] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:04.932113 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 45073us
I20260317 21:19:04.932752 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]) successfully deleted
W20260317 21:19:04.933472 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 1c3beb9232d74e45ab713ae8241eec4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c3beb9232d74e45ab713ae8241eec4c already in progress: creating tablet
W20260317 21:19:04.934067 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 because the tablet was not found. No further retry: Not found: Tablet not found: 6ab5f3dc7836425d917bd09f1e6bfe90
W20260317 21:19:04.934657 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 2607b9978c32466eb6dc8b8acfb429ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2607b9978c32466eb6dc8b8acfb429ce already in progress: creating tablet
I20260317 21:19:04.938297 7462 tablet_bootstrap.cc:492] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:04.938707 7462 ts_tablet_manager.cc:1403] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.130s user 0.006s sys 0.004s
I20260317 21:19:04.948766 7791 ts_tablet_manager.cc:1929] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:04.949735 7791 log.cc:1199] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/e858a01bc4594cc58853ea3239a8dbad
I20260317 21:19:04.950846 7791 ts_tablet_manager.cc:1950] T e858a01bc4594cc58853ea3239a8dbad P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:04.952064 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 179) took 2978 ms. Trace:
I20260317 21:19:04.955229 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d2f7c1058f8245428f38fe3245621b75" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:04.956892 6971 leader_election.cc:343] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:19:04.957396 6971 leader_election.cc:304] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:04.958736 7994 raft_consensus.cc:2749] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:04.964486 7467 tablet_bootstrap.cc:654] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.969398 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 6cf1a0922ed54674aaff8b0f49751c13 (DEFAULT_TABLE table=default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.969717 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 (DEFAULT_TABLE table=default.Table_bc634ee1d0df4afcb027ea787dce81b5 [id=19d461771e63402d98c3803a6b7cbae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.970590 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf1a0922ed54674aaff8b0f49751c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.970821 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab5f3dc7836425d917bd09f1e6bfe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.969838 7459 tablet_bootstrap.cc:654] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:04.973160 7878 raft_consensus.cc:493] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:04.973615 7878 raft_consensus.cc:515] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.975443 7878 leader_election.cc:290] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:04.977901 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:04.979466 7264 leader_election.cc:343] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:04.975412 7937 raft_consensus.cc:515] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:04.977775 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:04.990428 7261 leader_election.cc:343] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:04.991346 7261 leader_election.cc:304] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:04.993357 7305 tablet_service.cc:1511] Processing CreateTablet for tablet eb3a61ed542741d88d89ef6b4c53e9e6 (DEFAULT_TABLE table=default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.994446 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3a61ed542741d88d89ef6b4c53e9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.997764 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 6cf1a0922ed54674aaff8b0f49751c13 (DEFAULT_TABLE table=default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:04.999214 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf1a0922ed54674aaff8b0f49751c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.000830 7869 ts_tablet_manager.cc:1929] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:05.001377 7869 log.cc:1199] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/d2f7c1058f8245428f38fe3245621b75
I20260317 21:19:05.001580 7878 raft_consensus.cc:2749] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:05.002442 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.004850 7937 leader_election.cc:290] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:05.006925 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 161849us
I20260317 21:19:05.007827 7459 tablet_bootstrap.cc:492] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:05.008235 7459 ts_tablet_manager.cc:1403] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.004s
I20260317 21:19:05.009052 6674 master_service.cc:954] ReplaceTablet: received request to replace tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:05.009359 8004 raft_consensus.cc:493] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.009816 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:05.009869 8004 raft_consensus.cc:515] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.010428 7038 raft_consensus.cc:3060] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.012434 8004 leader_election.cc:290] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.014070 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet f749358db0f74ef78d9eee23b72bfe3e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:04 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:05.014292 7154 tablet_service.cc:1511] Processing CreateTablet for tablet d51e3d4702724484b776a95eb5acf7ef (DEFAULT_TABLE table=default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.015304 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d51e3d4702724484b776a95eb5acf7ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.015738 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 (DEFAULT_TABLE table=default.Table_bc634ee1d0df4afcb027ea787dce81b5 [id=19d461771e63402d98c3803a6b7cbae7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.016696 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ab5f3dc7836425d917bd09f1e6bfe90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:04.953038 7791 rpcz_store.cc:276] 0317 21:19:01.973390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:01.974778 (+ 1388us) service_pool.cc:225] Handling call
0317 21:19:04.874497 (+2899719us) ts_tablet_manager.cc:1151] Deleting tablet e858a01bc4594cc58853ea3239a8dbad
0317 21:19:04.874896 (+ 399us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.926076 (+ 51180us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.948296 (+ 22220us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:04.951998 (+ 3702us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2899049}
I20260317 21:19:05.026739 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet e858a01bc4594cc58853ea3239a8dbad (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]) successfully deleted
I20260317 21:19:05.023695 7038 raft_consensus.cc:2468] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:05.032096 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 181) took 3025 ms. Trace:
I20260317 21:19:05.037571 7459 raft_consensus.cc:359] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.038268 7459 raft_consensus.cc:385] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.038551 7459 raft_consensus.cc:740] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.037886 7791 rpcz_store.cc:276] 0317 21:19:02.006515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.006640 (+ 125us) service_pool.cc:225] Handling call
0317 21:19:05.031752 (+3025112us) ts_tablet_manager.cc:1151] Deleting tablet 88eae1e70bf94ef3ae0aa64706dcb599
0317 21:19:05.031801 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.032042 (+ 241us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3024473}
I20260317 21:19:05.039481 7159 tablet_service.cc:1511] Processing CreateTablet for tablet eb3a61ed542741d88d89ef6b4c53e9e6 (DEFAULT_TABLE table=default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:05.040009 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 88eae1e70bf94ef3ae0aa64706dcb599 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 88eae1e70bf94ef3ae0aa64706dcb599 already in progress: creating tablet
I20260317 21:19:05.040565 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eb3a61ed542741d88d89ef6b4c53e9e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.040513 7459 consensus_queue.cc:260] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.052109 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 180) took 3046 ms. Trace:
I20260317 21:19:05.052531 7791 rpcz_store.cc:276] 0317 21:19:02.006090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.016339 (+ 10249us) service_pool.cc:225] Handling call
0317 21:19:05.048297 (+3031958us) ts_tablet_manager.cc:1151] Deleting tablet b2dccb915fad4666bd7978f8c416db44
0317 21:19:05.051804 (+ 3507us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.052049 (+ 245us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3027096}
W20260317 21:19:05.053922 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet b2dccb915fad4666bd7978f8c416db44 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b2dccb915fad4666bd7978f8c416db44 already in progress: creating tablet
I20260317 21:19:05.055794 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 183) took 2776 ms. Trace:
I20260317 21:19:05.057080 7791 rpcz_store.cc:276] 0317 21:19:02.279292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.279469 (+ 177us) service_pool.cc:225] Handling call
0317 21:19:05.055479 (+2776010us) ts_tablet_manager.cc:1151] Deleting tablet a4cee7952cf5428b98340789a48d2da2
0317 21:19:05.055520 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.055742 (+ 222us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2775283}
W20260317 21:19:05.065590 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet a4cee7952cf5428b98340789a48d2da2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a4cee7952cf5428b98340789a48d2da2 already in progress: creating tablet
I20260317 21:19:05.065704 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 69801721e2b7422eba08becf2bcecfa2 (DEFAULT_TABLE table=default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.066668 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69801721e2b7422eba08becf2bcecfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.066910 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 186) took 2448 ms. Trace:
I20260317 21:19:05.069830 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 69801721e2b7422eba08becf2bcecfa2 (DEFAULT_TABLE table=default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.071442 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69801721e2b7422eba08becf2bcecfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.072830 7791 rpcz_store.cc:276] 0317 21:19:02.740671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.741140 (+ 469us) service_pool.cc:225] Handling call
0317 21:19:05.066528 (+2325388us) ts_tablet_manager.cc:1151] Deleting tablet 7d73f093d0e8434ca4c41cef6dfe310c
0317 21:19:05.066565 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.066848 (+ 283us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2324723}
W20260317 21:19:05.074357 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 7d73f093d0e8434ca4c41cef6dfe310c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d73f093d0e8434ca4c41cef6dfe310c already in progress: creating tablet
I20260317 21:19:05.075695 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 187) took 2334 ms. Trace:
I20260317 21:19:05.076174 7791 rpcz_store.cc:276] 0317 21:19:02.741123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.752214 (+ 11091us) service_pool.cc:225] Handling call
0317 21:19:05.075310 (+2323096us) ts_tablet_manager.cc:1151] Deleting tablet 252f728d183f440a9ef42894508d5755
0317 21:19:05.075352 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.075641 (+ 289us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2322345}
W20260317 21:19:05.077430 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 252f728d183f440a9ef42894508d5755 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252f728d183f440a9ef42894508d5755 already in progress: creating tablet
I20260317 21:19:05.078676 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 190) took 2174 ms. Trace:
I20260317 21:19:05.078972 7083 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:05.079051 7791 rpcz_store.cc:276] 0317 21:19:02.904731 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.909213 (+ 4482us) service_pool.cc:225] Handling call
0317 21:19:05.078350 (+2169137us) ts_tablet_manager.cc:1151] Deleting tablet a978fa28a31b4d589031c77526b62b8e
0317 21:19:05.078394 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.078626 (+ 232us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2168517}
W20260317 21:19:05.080334 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet a978fa28a31b4d589031c77526b62b8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a978fa28a31b4d589031c77526b62b8e already in progress: creating tablet
I20260317 21:19:05.081457 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 192) took 1924 ms. Trace:
I20260317 21:19:05.081984 7459 ts_tablet_manager.cc:1434] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.073s user 0.000s sys 0.007s
I20260317 21:19:05.081754 7791 rpcz_store.cc:276] 0317 21:19:03.156967 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.175547 (+ 18580us) service_pool.cc:225] Handling call
0317 21:19:05.081101 (+1905554us) ts_tablet_manager.cc:1151] Deleting tablet 7fc38947219047b5abdf71af5554a987
0317 21:19:05.081144 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.081409 (+ 265us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1904895}
W20260317 21:19:05.083071 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 7fc38947219047b5abdf71af5554a987 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fc38947219047b5abdf71af5554a987 already in progress: creating tablet
I20260317 21:19:05.086131 7791 tablet_replica.cc:333] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:05.086894 7791 raft_consensus.cc:2243] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:05.087345 7791 pending_rounds.cc:70] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:05.087569 7791 pending_rounds.cc:77] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7265412478698762240 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_0b009e9d95134391a1d2d41e80db86c7" dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" new_extra_config { } }
I20260317 21:19:05.061524 7015 tablet_service.cc:1511] Processing CreateTablet for tablet f5027b892229491eb5a829cacd29a700 (DEFAULT_TABLE table=default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.089123 7791 raft_consensus.cc:2272] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.089515 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5027b892229491eb5a829cacd29a700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.091094 7791 ts_tablet_manager.cc:1916] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.094655 7374 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:05.095489 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:05.103117 7995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Bootstrap starting.
I20260317 21:19:05.107429 7459 tablet_bootstrap.cc:492] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:05.114631 7158 tablet_service.cc:1511] Processing CreateTablet for tablet f5027b892229491eb5a829cacd29a700 (DEFAULT_TABLE table=default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.114984 7120 leader_election.cc:304] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:05.115046 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb332f981fc84a178fadae2f4908bdde" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:05.115720 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5027b892229491eb5a829cacd29a700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.115705 7038 raft_consensus.cc:2468] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:05.117225 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 6cf1a0922ed54674aaff8b0f49751c13 (DEFAULT_TABLE table=default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.118557 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6cf1a0922ed54674aaff8b0f49751c13. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.121511 7152 tablet_service.cc:1511] Processing CreateTablet for tablet 69801721e2b7422eba08becf2bcecfa2 (DEFAULT_TABLE table=default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.122927 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 69801721e2b7422eba08becf2bcecfa2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.127892 7462 raft_consensus.cc:359] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.132920 7937 raft_consensus.cc:2804] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:05.143326 7937 raft_consensus.cc:697] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:05.145087 7937 consensus_queue.cc:237] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.146684 7459 tablet_bootstrap.cc:654] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.146837 7791 ts_tablet_manager.cc:1929] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260317 21:19:05.147140 7791 log.cc:1199] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/2562cc2a3bb2430981c88475f9e1daa1
I20260317 21:19:05.147928 7791 ts_tablet_manager.cc:1950] T 2562cc2a3bb2430981c88475f9e1daa1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:05.150277 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 198) took 1864 ms. Trace:
I20260317 21:19:05.150645 7791 rpcz_store.cc:276] 0317 21:19:03.286000 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.286176 (+ 176us) service_pool.cc:225] Handling call
0317 21:19:05.085977 (+1799801us) ts_tablet_manager.cc:1151] Deleting tablet 2562cc2a3bb2430981c88475f9e1daa1
0317 21:19:05.086022 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.131769 (+ 45747us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.146597 (+ 14828us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.150198 (+ 3601us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1799223}
I20260317 21:19:05.152021 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]) successfully deleted
I20260317 21:19:05.153301 7791 tablet_replica.cc:333] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:05.154112 7791 raft_consensus.cc:2243] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:05.154937 7791 pending_rounds.cc:70] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:05.155220 7791 pending_rounds.cc:77] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412477625622528 op_type: NO_OP noop_request { }
I20260317 21:19:05.155273 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_aef4da11f840465eb7fc80e6423d7454" } modify_external_catalogs: true
I20260317 21:19:05.155579 7791 raft_consensus.cc:2889] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:05.155962 7791 raft_consensus.cc:2272] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.156016 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_aef4da11f840465eb7fc80e6423d7454" } modify_external_catalogs: true
I20260317 21:19:05.157922 7791 ts_tablet_manager.cc:1916] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.168686 7459 tablet_bootstrap.cc:492] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:05.169148 7459 ts_tablet_manager.cc:1403] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.062s user 0.013s sys 0.000s
I20260317 21:19:05.171835 7459 raft_consensus.cc:359] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.179289 7462 raft_consensus.cc:385] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.179698 7462 raft_consensus.cc:740] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.180500 7462 consensus_queue.cc:260] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.189361 7869 ts_tablet_manager.cc:1950] T d2f7c1058f8245428f38fe3245621b75 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
W20260317 21:19:05.189720 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 241678us
I20260317 21:19:05.190586 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 194) took 2688 ms. Trace:
I20260317 21:19:05.191076 7869 rpcz_store.cc:276] 0317 21:19:02.502055 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.564636 (+ 62581us) service_pool.cc:225] Handling call
0317 21:19:04.482186 (+1917550us) ts_tablet_manager.cc:1151] Deleting tablet d2f7c1058f8245428f38fe3245621b75
0317 21:19:04.485732 (+ 3546us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:04.847897 (+362165us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.000469 (+152572us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.190495 (+190026us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1796773}
I20260317 21:19:05.192101 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e4522181cc434203ac14c9b07587fec8" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:05.192755 7326 raft_consensus.cc:3060] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.193583 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 195) took 2626 ms. Trace:
I20260317 21:19:05.195914 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet d2f7c1058f8245428f38fe3245621b75 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:19:05.172508 7459 raft_consensus.cc:385] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.199400 7459 raft_consensus.cc:740] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.200042 7459 consensus_queue.cc:260] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.202242 7326 raft_consensus.cc:2468] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:05.202800 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.213994 7459 ts_tablet_manager.cc:1434] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.045s user 0.004s sys 0.001s
I20260317 21:19:05.216070 7791 ts_tablet_manager.cc:1929] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:05.216572 7791 log.cc:1199] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/82f8dd260a134810a50465e4a77848ca
I20260317 21:19:05.217397 7791 ts_tablet_manager.cc:1950] T 82f8dd260a134810a50465e4a77848ca P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:05.218250 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 200) took 1718 ms. Trace:
I20260317 21:19:05.218614 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4c8c24e29f304d99b8a6540fc42b887a" } new_table_name: "default.Table_eb6dce2e8eef477681d82e18ab326537" modify_external_catalogs: true
I20260317 21:19:05.222383 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:05.228067 7459 tablet_bootstrap.cc:492] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:05.218644 7791 rpcz_store.cc:276] 0317 21:19:03.500369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.500633 (+ 264us) service_pool.cc:225] Handling call
0317 21:19:05.153098 (+1652465us) ts_tablet_manager.cc:1151] Deleting tablet 82f8dd260a134810a50465e4a77848ca
0317 21:19:05.153158 (+ 60us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.166943 (+ 13785us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.215837 (+ 48894us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.218186 (+ 2349us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1651858}
I20260317 21:19:05.232821 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 82f8dd260a134810a50465e4a77848ca (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]) successfully deleted
I20260317 21:19:05.233767 7791 tablet_replica.cc:333] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:05.234493 7791 raft_consensus.cc:2243] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:05.235015 7791 raft_consensus.cc:2272] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.236907 7791 ts_tablet_manager.cc:1916] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.237643 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3bfb11e1f4714c45b0b46b83c6eac620"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:05.238631 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bcc665be09a84bbbb32857f85e717c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:05.239275 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3bfb11e1f4714c45b0b46b83c6eac620 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:05.240129 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcc665be09a84bbbb32857f85e717c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:05.241089 6668 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_3ab39e76bb8c42b8b4917eaf4ab5a2d9" } new_table_name: "default.Table_b4a0af0137f043219343ba24745f5077" modify_external_catalogs: true
I20260317 21:19:05.242278 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_445e903e8c9b4d39a1d294b6f91d57ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:05.243908 7462 ts_tablet_manager.cc:1434] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.305s user 0.007s sys 0.000s
W20260317 21:19:05.243856 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_445e903e8c9b4d39a1d294b6f91d57ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:05.245054 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_20aa87dfa03e46f5b2db253720a9e012"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:05.193964 7869 rpcz_store.cc:276] 0317 21:19:02.567389 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.567500 (+ 111us) service_pool.cc:225] Handling call
0317 21:19:05.193104 (+2625604us) ts_tablet_manager.cc:1151] Deleting tablet 7d73f093d0e8434ca4c41cef6dfe310c
0317 21:19:05.193171 (+ 67us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.193522 (+ 351us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2507480}
W20260317 21:19:05.246693 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20aa87dfa03e46f5b2db253720a9e012 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:05.246973 7878 raft_consensus.cc:493] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.247782 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 196) took 2679 ms. Trace:
I20260317 21:19:05.247527 7878 raft_consensus.cc:515] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.197165 7120 leader_election.cc:304] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:05.249857 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 52985.1us
I20260317 21:19:05.250352 8008 raft_consensus.cc:2804] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:05.250769 8008 raft_consensus.cc:493] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:05.251168 8008 raft_consensus.cc:3060] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:05.251901 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 7d73f093d0e8434ca4c41cef6dfe310c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7d73f093d0e8434ca4c41cef6dfe310c already in progress: creating tablet
I20260317 21:19:05.251962 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a662b365ec4846bf9e9ee243d41cb91a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:05.252983 7119 leader_election.cc:343] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: eb332f981fc84a178fadae2f4908bdde
W20260317 21:19:05.253887 7261 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.254532 7878 leader_election.cc:290] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:05.256134 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a662b365ec4846bf9e9ee243d41cb91a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:05.256286 7303 tablet_service.cc:1511] Processing CreateTablet for tablet f5027b892229491eb5a829cacd29a700 (DEFAULT_TABLE table=default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.210772 7467 tablet_bootstrap.cc:492] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
W20260317 21:19:05.257253 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 56607.4us
I20260317 21:19:05.257573 7467 ts_tablet_manager.cc:1403] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.570s user 0.015s sys 0.004s
W20260317 21:19:05.257583 7264 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.257958 7264 leader_election.cc:304] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:05.258100 7119 consensus_peers.cc:597] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:05.259086 7878 raft_consensus.cc:2749] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:05.267135 7229 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:05.275786 7008 tablet_service.cc:1511] Processing CreateTablet for tablet 829bc3959a9e4c2dbb6277cde5969b0e (DEFAULT_TABLE table=default.Table_4e0de322dbf141f38753722166148d81 [id=2652ee9fdb984f14a138c570bda4636d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.280678 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829bc3959a9e4c2dbb6277cde5969b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.283533 6675 catalog_manager.cc:5671] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:05.285567 6675 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:05.287652 7467 raft_consensus.cc:359] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.290925 7937 raft_consensus.cc:493] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.291610 7937 raft_consensus.cc:515] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.294804 7937 leader_election.cc:290] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.294687 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eee8b5588c24661889c593797c94269" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:05.295286 7038 raft_consensus.cc:2468] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:05.296545 7120 leader_election.cc:304] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:05.298095 7119 leader_election.cc:343] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 0eee8b5588c24661889c593797c94269
I20260317 21:19:05.298920 7937 raft_consensus.cc:2804] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:05.299311 7937 raft_consensus.cc:493] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:05.299721 7083 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:05.299659 7467 raft_consensus.cc:385] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.300985 7467 raft_consensus.cc:740] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.301829 7467 consensus_queue.cc:260] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.300716 7937 raft_consensus.cc:3060] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.304826 7459 tablet_bootstrap.cc:654] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.317518 7791 ts_tablet_manager.cc:1929] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:05.248489 7869 rpcz_store.cc:276] 0317 21:19:02.568221 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.568332 (+ 111us) service_pool.cc:225] Handling call
0317 21:19:05.247208 (+2678876us) ts_tablet_manager.cc:1151] Deleting tablet 252f728d183f440a9ef42894508d5755
0317 21:19:05.247279 (+ 71us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.247709 (+ 430us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2561320}
I20260317 21:19:05.325619 7467 ts_tablet_manager.cc:1434] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.068s user 0.006s sys 0.000s
I20260317 21:19:05.317919 7791 log.cc:1199] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/c6d5385171364a078d968bb0a489383a
I20260317 21:19:05.325783 7869 tablet_replica.cc:333] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:05.326606 7467 tablet_bootstrap.cc:492] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:05.327288 7791 ts_tablet_manager.cc:1950] T c6d5385171364a078d968bb0a489383a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:05.328744 7878 raft_consensus.cc:493] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.329200 7878 raft_consensus.cc:515] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:05.329996 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 252f728d183f440a9ef42894508d5755 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 252f728d183f440a9ef42894508d5755 already in progress: creating tablet
I20260317 21:19:05.332131 7878 leader_election.cc:290] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:05.332022 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "275e9cd7f14040d98eb85a6bb1010c2c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:05.333258 7264 leader_election.cc:343] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.333735 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "275e9cd7f14040d98eb85a6bb1010c2c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:05.334417 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 201) took 1714 ms. Trace:
I20260317 21:19:05.334615 7459 tablet_bootstrap.cc:492] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:05.334122 8008 raft_consensus.cc:515] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.334987 7459 ts_tablet_manager.cc:1403] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.107s user 0.006s sys 0.003s
W20260317 21:19:05.335377 7261 leader_election.cc:343] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.335836 7261 leader_election.cc:304] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:05.334765 7791 rpcz_store.cc:276] 0317 21:19:03.620736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.621328 (+ 592us) service_pool.cc:225] Handling call
0317 21:19:05.233607 (+1612279us) ts_tablet_manager.cc:1151] Deleting tablet c6d5385171364a078d968bb0a489383a
0317 21:19:05.233664 (+ 57us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.288356 (+ 54692us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.317284 (+ 28928us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.334358 (+ 17074us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1611608}
I20260317 21:19:05.337059 7878 raft_consensus.cc:2749] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:05.337654 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet c6d5385171364a078d968bb0a489383a (table default.Table_4493a733fd0a496885b7745fcfc6cf2e [id=e8d1164df252476a82baf64ec8c6c9cf]) successfully deleted
I20260317 21:19:05.337507 7459 raft_consensus.cc:359] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.338122 7459 raft_consensus.cc:385] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.338380 7459 raft_consensus.cc:740] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.342401 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb332f981fc84a178fadae2f4908bdde" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:05.342993 7038 raft_consensus.cc:3060] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.343364 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 202) took 1722 ms. Trace:
I20260317 21:19:05.343760 7791 rpcz_store.cc:276] 0317 21:19:03.621128 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.675658 (+ 54530us) service_pool.cc:225] Handling call
0317 21:19:05.342928 (+1667270us) ts_tablet_manager.cc:1151] Deleting tablet c9b7f7c864004b9abde0a0201e196165
0317 21:19:05.342975 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.343242 (+ 267us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1637984}
W20260317 21:19:05.344967 7119 leader_election.cc:343] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: eb332f981fc84a178fadae2f4908bdde
W20260317 21:19:05.344971 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet c9b7f7c864004b9abde0a0201e196165 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7f7c864004b9abde0a0201e196165 already in progress: creating tablet
I20260317 21:19:05.326606 7869 raft_consensus.cc:2243] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:05.345944 7869 raft_consensus.cc:2272] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.348412 7791 tablet_replica.cc:333] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:05.348937 8008 leader_election.cc:290] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.349541 7791 raft_consensus.cc:2243] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:05.350059 7791 raft_consensus.cc:2272] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.351637 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:05.351997 7791 ts_tablet_manager.cc:1916] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.352178 7937 raft_consensus.cc:515] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.353821 7038 raft_consensus.cc:2468] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:05.348124 7869 ts_tablet_manager.cc:1916] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.362885 7120 leader_election.cc:304] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:05.364696 8008 raft_consensus.cc:2804] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:05.364979 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0eee8b5588c24661889c593797c94269" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:05.365192 8008 raft_consensus.cc:697] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:05.365542 7035 raft_consensus.cc:3060] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.365355 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 829bc3959a9e4c2dbb6277cde5969b0e (DEFAULT_TABLE table=default.Table_4e0de322dbf141f38753722166148d81 [id=2652ee9fdb984f14a138c570bda4636d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.365980 8008 consensus_queue.cc:237] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:05.367251 7119 leader_election.cc:343] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 0eee8b5588c24661889c593797c94269
I20260317 21:19:05.361342 7459 consensus_queue.cc:260] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.368332 7937 leader_election.cc:290] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.374612 7459 ts_tablet_manager.cc:1434] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.039s user 0.004s sys 0.003s
I20260317 21:19:05.375531 7459 tablet_bootstrap.cc:492] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:05.377038 7791 ts_tablet_manager.cc:1929] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:05.377384 7791 log.cc:1199] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/935faf7f9129440a91635bfa96191cd6
I20260317 21:19:05.358389 7467 tablet_bootstrap.cc:654] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.379227 7035 raft_consensus.cc:2468] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:05.380508 7120 leader_election.cc:304] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:05.393718 7937 raft_consensus.cc:2804] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:05.394330 7937 raft_consensus.cc:697] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:05.395109 7791 ts_tablet_manager.cc:1950] T 935faf7f9129440a91635bfa96191cd6 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:05.395219 7937 consensus_queue.cc:237] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.395969 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 203) took 1603 ms. Trace:
I20260317 21:19:05.396276 7791 rpcz_store.cc:276] 0317 21:19:03.792672 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.792855 (+ 183us) service_pool.cc:225] Handling call
0317 21:19:05.348233 (+1555378us) ts_tablet_manager.cc:1151] Deleting tablet 935faf7f9129440a91635bfa96191cd6
0317 21:19:05.348278 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.359447 (+ 11169us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.376817 (+ 17370us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.395915 (+ 19098us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1554637}
I20260317 21:19:05.397560 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 935faf7f9129440a91635bfa96191cd6 (table default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]) successfully deleted
I20260317 21:19:05.400272 7462 tablet_bootstrap.cc:492] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:05.401032 7791 tablet_replica.cc:333] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:05.401937 7791 raft_consensus.cc:2243] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:05.403065 7791 raft_consensus.cc:2272] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.405803 7791 ts_tablet_manager.cc:1916] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:05.406445 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 164063us
I20260317 21:19:05.412094 7459 tablet_bootstrap.cc:654] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.415041 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f5027b892229491eb5a829cacd29a700. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.428728 6667 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 6f0b60f409b34a2ba237c6aaf529c238
I20260317 21:19:05.431035 6674 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e
I20260317 21:19:05.432369 6674 catalog_manager.cc:6741] ReplaceTablet: tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e deleted and replaced by tablet e988430ec1074531b0e173e96983a0cb
I20260317 21:19:05.436208 7791 ts_tablet_manager.cc:1929] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:05.436589 7791 log.cc:1199] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/d2f7c1058f8245428f38fe3245621b75
I20260317 21:19:05.437654 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e with delete_type TABLET_DATA_DELETED (replaced by tablet e988430ec1074531b0e173e96983a0cb) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:05.446416 7791 ts_tablet_manager.cc:1950] T d2f7c1058f8245428f38fe3245621b75 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:05.447201 7869 ts_tablet_manager.cc:1929] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:05.447594 7869 log.cc:1199] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/dc929bae186841d2aba674659e457301
I20260317 21:19:05.448253 7869 ts_tablet_manager.cc:1950] T dc929bae186841d2aba674659e457301 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:05.448293 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 204) took 1399 ms. Trace:
I20260317 21:19:05.448585 7791 rpcz_store.cc:276] 0317 21:19:04.048969 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.049111 (+ 142us) service_pool.cc:225] Handling call
0317 21:19:05.400879 (+1351768us) ts_tablet_manager.cc:1151] Deleting tablet d2f7c1058f8245428f38fe3245621b75
0317 21:19:05.400922 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.427900 (+ 26978us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.435981 (+ 8081us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.448234 (+ 12253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1299276}
I20260317 21:19:05.450242 7467 tablet_bootstrap.cc:492] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:05.450695 7467 ts_tablet_manager.cc:1403] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.124s user 0.008s sys 0.003s
I20260317 21:19:05.450106 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 208) took 1329 ms. Trace:
I20260317 21:19:05.452288 7791 rpcz_store.cc:276] 0317 21:19:04.121175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.121332 (+ 157us) service_pool.cc:225] Handling call
0317 21:19:05.449785 (+1328453us) ts_tablet_manager.cc:1151] Deleting tablet 16f64073188f4d06859e1a701df73242
0317 21:19:05.449834 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.450059 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1327810}
I20260317 21:19:05.453608 7791 tablet_replica.cc:333] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:05.453356 7467 raft_consensus.cc:359] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.454041 7467 raft_consensus.cc:385] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.454308 7467 raft_consensus.cc:740] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.454420 7791 raft_consensus.cc:2243] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:05.455073 7791 pending_rounds.cc:70] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:05.455284 7791 pending_rounds.cc:77] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412484138991616 op_type: NO_OP noop_request { }
I20260317 21:19:05.455616 7791 raft_consensus.cc:2889] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:05.455950 7791 raft_consensus.cc:2272] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.458176 7791 ts_tablet_manager.cc:1916] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.466879 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet d2f7c1058f8245428f38fe3245621b75 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]) successfully deleted
I20260317 21:19:05.464807 7459 tablet_bootstrap.cc:492] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:05.470332 7459 ts_tablet_manager.cc:1403] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.095s user 0.010s sys 0.000s
I20260317 21:19:05.472528 7459 raft_consensus.cc:359] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.473165 7459 raft_consensus.cc:385] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.473394 7459 raft_consensus.cc:740] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.473946 7459 consensus_queue.cc:260] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.467139 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e with delete_type TABLET_DATA_DELETED (replaced by tablet e988430ec1074531b0e173e96983a0cb) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.475914 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829bc3959a9e4c2dbb6277cde5969b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.481523 7459 ts_tablet_manager.cc:1434] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.011s user 0.006s sys 0.000s
I20260317 21:19:05.482532 7459 tablet_bootstrap.cc:492] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:05.489333 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 198) took 2716 ms. Trace:
I20260317 21:19:05.489851 7869 rpcz_store.cc:276] 0317 21:19:02.827615 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:02.827758 (+ 143us) service_pool.cc:225] Handling call
0317 21:19:05.325585 (+2497827us) ts_tablet_manager.cc:1151] Deleting tablet dc929bae186841d2aba674659e457301
0317 21:19:05.325660 (+ 75us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.394526 (+ 68866us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.446946 (+ 52420us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.489271 (+ 42325us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27520,"tablet-delete.queue_time_us":2497189}
I20260317 21:19:05.491731 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 202) took 1969 ms. Trace:
I20260317 21:19:05.492143 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 275e9cd7f14040d98eb85a6bb1010c2c (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]). Expected version 1 got 0
I20260317 21:19:05.492615 7869 rpcz_store.cc:276] 0317 21:19:03.522631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.524918 (+ 2287us) service_pool.cc:225] Handling call
0317 21:19:05.491362 (+1966444us) ts_tablet_manager.cc:1151] Deleting tablet 7fc38947219047b5abdf71af5554a987
0317 21:19:05.491414 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.491675 (+ 261us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1965664}
I20260317 21:19:05.493721 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet dc929bae186841d2aba674659e457301 (table default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]) successfully deleted
W20260317 21:19:05.494514 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 7fc38947219047b5abdf71af5554a987 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7fc38947219047b5abdf71af5554a987 already in progress: creating tablet
I20260317 21:19:05.495803 7467 consensus_queue.cc:260] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.497331 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet ae1b2a549af6485ca3faee3f0ad845a2 from {username='slave'} at 127.0.0.1:39038
W20260317 21:19:05.501868 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 16f64073188f4d06859e1a701df73242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f64073188f4d06859e1a701df73242 already in progress: creating tablet
I20260317 21:19:05.507333 7462 tablet_bootstrap.cc:654] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.507417 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 932a5eb6cc2848f6a7d7c02ea47e5100 (DEFAULT_TABLE table=default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.510612 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932a5eb6cc2848f6a7d7c02ea47e5100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.511348 7994 raft_consensus.cc:493] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.511933 7994 raft_consensus.cc:515] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.517186 7878 raft_consensus.cc:493] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.517652 7878 raft_consensus.cc:515] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.518683 7866 ts_tablet_manager.cc:1929] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260317 21:19:05.519037 7866 log.cc:1199] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/f749358db0f74ef78d9eee23b72bfe3e
I20260317 21:19:05.519747 7866 ts_tablet_manager.cc:1950] T f749358db0f74ef78d9eee23b72bfe3e P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:05.520900 7878 leader_election.cc:290] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:05.526784 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e with delete_type TABLET_DATA_DELETED (replaced by tablet e988430ec1074531b0e173e96983a0cb) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:05.534348 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 829bc3959a9e4c2dbb6277cde5969b0e (DEFAULT_TABLE table=default.Table_4e0de322dbf141f38753722166148d81 [id=2652ee9fdb984f14a138c570bda4636d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.535462 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 829bc3959a9e4c2dbb6277cde5969b0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.545099 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 932a5eb6cc2848f6a7d7c02ea47e5100 (DEFAULT_TABLE table=default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.548453 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932a5eb6cc2848f6a7d7c02ea47e5100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.561041 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 7b7b54cbff0c4fd5813d197ddbc899cc (DEFAULT_TABLE table=default.Table_b78a8f62954143d9b88c4fa88d44cbf2 [id=b816766f31714663b685cfcc9a9f537a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.562510 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b7b54cbff0c4fd5813d197ddbc899cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.568895 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 201) took 2053 ms. Trace:
I20260317 21:19:05.570004 7869 rpcz_store.cc:276] 0317 21:19:03.515129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.521870 (+ 6741us) service_pool.cc:225] Handling call
0317 21:19:05.549190 (+2027320us) ts_tablet_manager.cc:1151] Deleting tablet a978fa28a31b4d589031c77526b62b8e
0317 21:19:05.561830 (+ 12640us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.568822 (+ 6992us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2006537}
I20260317 21:19:05.571156 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:05.572863 7261 leader_election.cc:343] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.579269 7869 tablet_replica.cc:333] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:05.581566 7869 raft_consensus.cc:2243] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:05.582791 7869 pending_rounds.cc:70] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:19:05.585775 7459 tablet_bootstrap.cc:654] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.585892 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:05.586699 7467 ts_tablet_manager.cc:1434] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.136s user 0.006s sys 0.001s
W20260317 21:19:05.588233 7264 leader_election.cc:343] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.588765 7264 leader_election.cc:304] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:05.589408 7994 leader_election.cc:290] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.590185 7878 raft_consensus.cc:2749] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:05.590657 7994 raft_consensus.cc:493] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.591075 7994 raft_consensus.cc:515] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.597729 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:05.598479 7326 raft_consensus.cc:2468] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:05.598654 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 932a5eb6cc2848f6a7d7c02ea47e5100 (DEFAULT_TABLE table=default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.599903 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 7b7b54cbff0c4fd5813d197ddbc899cc (DEFAULT_TABLE table=default.Table_b78a8f62954143d9b88c4fa88d44cbf2 [id=b816766f31714663b685cfcc9a9f537a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.601742 7016 tablet_service.cc:1511] Processing CreateTablet for tablet b9d1076839894509805be59b751aa847 (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.605196 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 932a5eb6cc2848f6a7d7c02ea47e5100. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.606462 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d1076839894509805be59b751aa847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.609005 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b7b54cbff0c4fd5813d197ddbc899cc. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:05.611353 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet a978fa28a31b4d589031c77526b62b8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a978fa28a31b4d589031c77526b62b8e already in progress: creating tablet
I20260317 21:19:05.611914 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:05.612486 7180 raft_consensus.cc:2468] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:05.613313 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:05.614068 7183 raft_consensus.cc:2393] T d05973b63a8c445995ca51d8aec7bcf9 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:05.615383 6971 leader_election.cc:304] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:05.616489 8025 raft_consensus.cc:2804] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:05.616900 8025 raft_consensus.cc:493] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:05.617218 8025 raft_consensus.cc:3060] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.624970 8025 raft_consensus.cc:515] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.625826 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 5314c562bc9d457a8e1f4930e8d3a7ff
W20260317 21:19:05.627801 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ff04afc5dc834ac4a51f97e29e4cc96e has no consensus state
I20260317 21:19:05.629113 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:05.629693 7183 raft_consensus.cc:3060] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.633343 7467 tablet_bootstrap.cc:492] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:05.636322 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.637624 7183 raft_consensus.cc:2468] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:05.639611 6971 leader_election.cc:304] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:05.641148 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5e080308f93448138a4846ab6bec7dfa (table default.Table_0d594c7cc868438fa7b25513ce01f0e9 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e080308f93448138a4846ab6bec7dfa has no consensus state
I20260317 21:19:05.641258 8028 raft_consensus.cc:2804] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:05.646907 8025 leader_election.cc:290] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.648053 7462 tablet_bootstrap.cc:492] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:05.648389 8028 raft_consensus.cc:697] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:05.649601 8028 consensus_queue.cc:237] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.648511 7462 ts_tablet_manager.cc:1403] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.248s user 0.010s sys 0.001s
I20260317 21:19:05.661093 7994 leader_election.cc:290] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:05.664191 6668 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d69e14125d7d4412869c20759acc2336 has no consensus state
I20260317 21:19:05.676111 6669 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 275e9cd7f14040d98eb85a6bb1010c2c (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]). Expected version 1 got 0
I20260317 21:19:05.676550 7459 tablet_bootstrap.cc:492] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:05.676993 7459 ts_tablet_manager.cc:1403] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.195s user 0.009s sys 0.004s
I20260317 21:19:05.683429 7462 raft_consensus.cc:359] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.684295 7462 raft_consensus.cc:385] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.684671 7462 raft_consensus.cc:740] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.686743 7035 raft_consensus.cc:3060] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.690780 7459 raft_consensus.cc:359] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.691419 7459 raft_consensus.cc:385] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.699261 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:05.583104 7869 pending_rounds.cc:77] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7265412478698762240 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "2562cc2a3bb2430981c88475f9e1daa1" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_0b009e9d95134391a1d2d41e80db86c7" dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" new_extra_config { } }
I20260317 21:19:05.702405 7869 raft_consensus.cc:2272] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:05.711983 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 054535edcd5041e9b0429c3e7248b1b7 (DEFAULT_TABLE table=default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.712025 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 054535edcd5041e9b0429c3e7248b1b7 (DEFAULT_TABLE table=default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.713100 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054535edcd5041e9b0429c3e7248b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.713110 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054535edcd5041e9b0429c3e7248b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:05.714973 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 55087.6us
I20260317 21:19:05.691684 7459 raft_consensus.cc:740] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:05.719062 7459 consensus_queue.cc:260] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.620265 7152 tablet_service.cc:1511] Processing CreateTablet for tablet 7b7b54cbff0c4fd5813d197ddbc899cc (DEFAULT_TABLE table=default.Table_b78a8f62954143d9b88c4fa88d44cbf2 [id=b816766f31714663b685cfcc9a9f537a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.698544 8008 raft_consensus.cc:493] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.723491 7995 log.cc:826] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Log is configured to *not* fsync() on all Append() calls
I20260317 21:19:05.725411 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7b7b54cbff0c4fd5813d197ddbc899cc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.727670 7459 ts_tablet_manager.cc:1434] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.050s user 0.005s sys 0.001s
I20260317 21:19:05.728709 7459 tablet_bootstrap.cc:492] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:05.732964 7159 tablet_service.cc:1511] Processing CreateTablet for tablet b9d1076839894509805be59b751aa847 (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.734099 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d1076839894509805be59b751aa847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.704406 7869 ts_tablet_manager.cc:1916] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:05.732297 8008 raft_consensus.cc:515] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.746580 8008 leader_election.cc:290] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.752571 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 0dd4c1efc0624957832c17dc7a4db3f3 (DEFAULT_TABLE table=default.Table_687dd24cc64a425b80bcfb73ef08d1fd [id=e8f700fb229d465ca87459fd70770227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.753875 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd4c1efc0624957832c17dc7a4db3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.754926 7459 tablet_bootstrap.cc:654] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.754941 6675 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
I20260317 21:19:05.765607 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet ae1b2a549af6485ca3faee3f0ad845a2
I20260317 21:19:05.766944 6667 catalog_manager.cc:6741] ReplaceTablet: tablet ae1b2a549af6485ca3faee3f0ad845a2 deleted and replaced by tablet 43702e625cb147339bd5871a5ee83e17
I20260317 21:19:05.768714 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet ae1b2a549af6485ca3faee3f0ad845a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 43702e625cb147339bd5871a5ee83e17) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:05.770486 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0a9cb20926ea45aaa419518e3f9c57ab" } modify_external_catalogs: true
I20260317 21:19:05.771013 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0a9cb20926ea45aaa419518e3f9c57ab" } modify_external_catalogs: true
I20260317 21:19:05.783707 7791 ts_tablet_manager.cc:1929] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:05.784134 7791 log.cc:1199] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/dc929bae186841d2aba674659e457301
I20260317 21:19:05.807636 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.809998 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet ae1b2a549af6485ca3faee3f0ad845a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 43702e625cb147339bd5871a5ee83e17) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.812479 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 0dd4c1efc0624957832c17dc7a4db3f3 (DEFAULT_TABLE table=default.Table_687dd24cc64a425b80bcfb73ef08d1fd [id=e8f700fb229d465ca87459fd70770227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.813637 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd4c1efc0624957832c17dc7a4db3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.821950 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 00f34026fd2f4575bb86a9dadb733fe0
I20260317 21:19:05.824712 7462 consensus_queue.cc:260] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.825394 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564a02888d854ea58d7a590cba816b86" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:05.826540 7328 raft_consensus.cc:3060] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.821065 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d05973b63a8c445995ca51d8aec7bcf9" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:05.830124 7119 leader_election.cc:343] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05
I20260317 21:19:05.810840 8036 consensus_queue.cc:1048] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:05.815307 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:05.816413 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet ae1b2a549af6485ca3faee3f0ad845a2 with delete_type TABLET_DATA_DELETED (replaced by tablet 43702e625cb147339bd5871a5ee83e17) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:05.817127 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 054535edcd5041e9b0429c3e7248b1b7 (DEFAULT_TABLE table=default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.835680 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 054535edcd5041e9b0429c3e7248b1b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.835388 7878 consensus_queue.cc:1048] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:05.839203 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:05.818416 7303 tablet_service.cc:1511] Processing CreateTablet for tablet b9d1076839894509805be59b751aa847 (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.841396 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9d1076839894509805be59b751aa847. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.841681 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
W20260317 21:19:05.871222 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 67487.7us
I20260317 21:19:05.871999 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]) successfully deleted
W20260317 21:19:05.872763 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e already in progress: creating tablet
W20260317 21:19:05.873456 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 5314c562bc9d457a8e1f4930e8d3a7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5314c562bc9d457a8e1f4930e8d3a7ff already in progress: creating tablet
W20260317 21:19:05.874092 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet ae1b2a549af6485ca3faee3f0ad845a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae1b2a549af6485ca3faee3f0ad845a2 already in progress: creating tablet
I20260317 21:19:05.875422 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:05.878943 7154 tablet_service.cc:1511] Processing CreateTablet for tablet cef932d50d4544ad9c5ca245bc006f29 (DEFAULT_TABLE table=default.Table_bd7dbccbd72148e7ae945d5fded9e16a [id=d3c6b27b8483401eb16133bf6c718ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.882498 8036 raft_consensus.cc:493] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.883280 8036 raft_consensus.cc:515] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.890307 7791 ts_tablet_manager.cc:1950] T dc929bae186841d2aba674659e457301 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:05.890702 6973 leader_election.cc:304] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:05.891578 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 209) took 1594 ms. Trace:
I20260317 21:19:05.892079 7791 rpcz_store.cc:276] 0317 21:19:04.297265 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.297505 (+ 240us) service_pool.cc:225] Handling call
0317 21:19:05.453460 (+1155955us) ts_tablet_manager.cc:1151] Deleting tablet dc929bae186841d2aba674659e457301
0317 21:19:05.453501 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.644739 (+191238us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.783465 (+138726us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:05.891513 (+108048us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1155212}
I20260317 21:19:05.893057 7994 raft_consensus.cc:2749] T d05973b63a8c445995ca51d8aec7bcf9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:05.913404 8028 raft_consensus.cc:493] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:05.926299 8028 raft_consensus.cc:515] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.927709 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet dc929bae186841d2aba674659e457301 (table default.Table_5f4dc71734d540319e6f0d19d4037ba8 [id=57272480ab6a48ce8126d81d700e3833]) successfully deleted
I20260317 21:19:05.913581 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35023195b9094d58827921a1dcb2ff05" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:05.929595 7038 raft_consensus.cc:2468] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:05.915987 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:05.917032 7010 tablet_service.cc:1511] Processing CreateTablet for tablet cef932d50d4544ad9c5ca245bc006f29 (DEFAULT_TABLE table=default.Table_bd7dbccbd72148e7ae945d5fded9e16a [id=d3c6b27b8483401eb16133bf6c718ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.921113 7016 tablet_service.cc:1511] Processing CreateTablet for tablet b8c9cd4f760f44c88cdd986542755443 (DEFAULT_TABLE table=default.Table_6bb2d165f08a40d7bd84414364358f7e [id=fb617cbcbd634696a101417011a3a53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.931653 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef932d50d4544ad9c5ca245bc006f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.932312 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c9cd4f760f44c88cdd986542755443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.945150 8028 leader_election.cc:290] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:05.946223 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 211) took 1546 ms. Trace:
W20260317 21:19:05.947954 6973 leader_election.cc:343] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05
I20260317 21:19:05.948231 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bd7dbccbd72148e7ae945d5fded9e16a" } modify_external_catalogs: true
I20260317 21:19:05.949080 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bd7dbccbd72148e7ae945d5fded9e16a" } modify_external_catalogs: true
I20260317 21:19:05.888461 7467 tablet_bootstrap.cc:654] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:05.913964 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef932d50d4544ad9c5ca245bc006f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.920001 7155 tablet_service.cc:1511] Processing CreateTablet for tablet b8c9cd4f760f44c88cdd986542755443 (DEFAULT_TABLE table=default.Table_6bb2d165f08a40d7bd84414364358f7e [id=fb617cbcbd634696a101417011a3a53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:05.956374 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c9cd4f760f44c88cdd986542755443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:05.960856 7459 tablet_bootstrap.cc:492] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:05.962064 6670 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b8f93eb9b7b047cf9b6604988438ec76" } new_table_name: "default.Table_aacec8c3d7ba49f9b6742baf37b423a8" modify_external_catalogs: true
I20260317 21:19:05.964043 7459 ts_tablet_manager.cc:1403] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.236s user 0.028s sys 0.030s
I20260317 21:19:05.932214 7229 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:05.952310 7120 leader_election.cc:304] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:05.970160 7937 raft_consensus.cc:2804] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:05.970542 7937 raft_consensus.cc:493] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:05.970885 7937 raft_consensus.cc:3060] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:05.974601 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_53241366a970456f98258014850933b1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:05.975128 7328 raft_consensus.cc:2468] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:05.976500 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_53241366a970456f98258014850933b1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:05.979297 7459 raft_consensus.cc:359] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:05.980016 7459 raft_consensus.cc:385] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:05.989442 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:05.991086 7264 leader_election.cc:343] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.994267 7462 ts_tablet_manager.cc:1434] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.337s user 0.013s sys 0.015s
I20260317 21:19:05.997327 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:05.997749 8036 leader_election.cc:290] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:05.999008 7261 leader_election.cc:343] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:05.999440 7261 leader_election.cc:304] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:05.946555 7791 rpcz_store.cc:276] 0317 21:19:04.399618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.399821 (+ 203us) service_pool.cc:225] Handling call
0317 21:19:05.945816 (+1545995us) ts_tablet_manager.cc:1151] Deleting tablet 1c3beb9232d74e45ab713ae8241eec4c
0317 21:19:05.945863 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:05.946154 (+ 291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1545196}
I20260317 21:19:06.000804 7459 raft_consensus.cc:740] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:06.001026 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 1c3beb9232d74e45ab713ae8241eec4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c3beb9232d74e45ab713ae8241eec4c already in progress: creating tablet
I20260317 21:19:06.001487 7459 consensus_queue.cc:260] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.003041 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 213) took 1528 ms. Trace:
I20260317 21:19:06.004431 7459 ts_tablet_manager.cc:1434] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260317 21:19:06.013926 8036 raft_consensus.cc:2749] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:06.023495 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0d594c7cc868438fa7b25513ce01f0e9" } new_table_name: "default.Table_7af8e07bbd5b40b8ad18551c8c575d60" modify_external_catalogs: true
I20260317 21:19:06.023998 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_760f5ff0097f4ae6aa93439b05ef00c4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:06.024901 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_f95f659a6c96464296d2b1b3bba4af14"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:06.025758 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_760f5ff0097f4ae6aa93439b05ef00c4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:06.027225 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f95f659a6c96464296d2b1b3bba4af14 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:06.027228 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_c194cf09401a41589629e3c00ffbaf15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:06.028849 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c194cf09401a41589629e3c00ffbaf15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:06.030766 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 78483.8us
I20260317 21:19:06.034065 7462 tablet_bootstrap.cc:492] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:06.045179 7462 tablet_bootstrap.cc:654] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.048215 7459 tablet_bootstrap.cc:492] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:06.062145 8028 raft_consensus.cc:493] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:06.062644 8028 raft_consensus.cc:515] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.064621 8028 leader_election.cc:290] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:06.064549 6667 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
W20260317 21:19:06.067178 6973 leader_election.cc:343] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 59fce28afbeb4d8eadf63f9a97d1183d
I20260317 21:19:06.066759 6667 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:06.081004 6671 master_service.cc:954] ReplaceTablet: received request to replace tablet b9d1076839894509805be59b751aa847 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:06.083623 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 0dd4c1efc0624957832c17dc7a4db3f3 (DEFAULT_TABLE table=default.Table_687dd24cc64a425b80bcfb73ef08d1fd [id=e8f700fb229d465ca87459fd70770227]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:06.084844 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0dd4c1efc0624957832c17dc7a4db3f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:06.084777 8036 raft_consensus.cc:493] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:06.085259 8036 raft_consensus.cc:515] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.090209 8036 leader_election.cc:290] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:06.096458 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:06.097808 7264 leader_election.cc:343] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.003949 7791 rpcz_store.cc:276] 0317 21:19:04.474277 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.474399 (+ 122us) service_pool.cc:225] Handling call
0317 21:19:06.002163 (+1527764us) ts_tablet_manager.cc:1151] Deleting tablet 2607b9978c32466eb6dc8b8acfb429ce
0317 21:19:06.002302 (+ 139us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.002905 (+ 603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1527048}
W20260317 21:19:06.102648 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 2607b9978c32466eb6dc8b8acfb429ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2607b9978c32466eb6dc8b8acfb429ce already in progress: creating tablet
I20260317 21:19:06.104979 7791 tablet_replica.cc:333] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:06.107438 7298 tablet_service.cc:1511] Processing CreateTablet for tablet cef932d50d4544ad9c5ca245bc006f29 (DEFAULT_TABLE table=default.Table_bd7dbccbd72148e7ae945d5fded9e16a [id=d3c6b27b8483401eb16133bf6c718ced]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:06.108635 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cef932d50d4544ad9c5ca245bc006f29. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:06.116991 8034 consensus_queue.cc:1048] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:06.121415 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:57 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:06.122304 7866 tablet_replica.cc:333] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:06.123036 7866 raft_consensus.cc:2243] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:06.123492 7866 raft_consensus.cc:2272] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:06.124547 7301 tablet_service.cc:1511] Processing CreateTablet for tablet b8c9cd4f760f44c88cdd986542755443 (DEFAULT_TABLE table=default.Table_6bb2d165f08a40d7bd84414364358f7e [id=fb617cbcbd634696a101417011a3a53a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:06.126142 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8c9cd4f760f44c88cdd986542755443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:06.128360 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35023195b9094d58827921a1dcb2ff05" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:06.129158 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "59fce28afbeb4d8eadf63f9a97d1183d" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:06.130260 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:06.131435 7866 ts_tablet_manager.cc:1916] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:06.131625 7261 leader_election.cc:343] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.132489 7261 leader_election.cc:304] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:06.143626 6971 leader_election.cc:343] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.144104 6971 leader_election.cc:304] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:06.145351 8028 raft_consensus.cc:2749] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:06.147814 7937 raft_consensus.cc:515] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.149127 7180 raft_consensus.cc:2393] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:19:06.152204 7467 tablet_bootstrap.cc:492] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:06.152745 7467 ts_tablet_manager.cc:1403] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.520s user 0.007s sys 0.005s
I20260317 21:19:06.155133 7467 raft_consensus.cc:359] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.160313 6971 leader_election.cc:304] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:06.161598 8028 raft_consensus.cc:3060] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:06.162959 7459 tablet_bootstrap.cc:654] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.171743 8028 raft_consensus.cc:2749] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:06.172915 8036 raft_consensus.cc:2749] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:06.191054 7791 raft_consensus.cc:2243] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:06.191570 7791 raft_consensus.cc:2272] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:06.193298 7791 ts_tablet_manager.cc:1916] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:06.194427 7866 ts_tablet_manager.cc:1929] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260317 21:19:06.194208 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 00f34026fd2f4575bb86a9dadb733fe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00f34026fd2f4575bb86a9dadb733fe0 already in progress: creating tablet
I20260317 21:19:06.199258 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "35023195b9094d58827921a1dcb2ff05" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:06.202123 7866 log.cc:1199] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:19:06.203503 7866 ts_tablet_manager.cc:1950] T bcd99ca9f6c540e48e29d8659fbfe777 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:06.212131 7462 tablet_bootstrap.cc:492] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:06.182634 8008 consensus_queue.cc:1048] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:06.214509 7120 consensus_peers.cc:597] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:06.216778 7937 leader_election.cc:290] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:06.218538 7119 leader_election.cc:343] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05
I20260317 21:19:06.222707 7467 raft_consensus.cc:385] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.223176 7467 raft_consensus.cc:740] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.234373 8028 raft_consensus.cc:493] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:06.235136 8028 raft_consensus.cc:515] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.237902 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b287c83b77548f19546927b05905746" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:06.238350 8028 leader_election.cc:290] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:06.238979 7152 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:06.239286 6973 leader_election.cc:343] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 2b287c83b77548f19546927b05905746
I20260317 21:19:06.243477 7995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 4 replicates
I20260317 21:19:06.244302 7869 ts_tablet_manager.cc:1929] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260317 21:19:06.244936 7995 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Bootstrap complete.
I20260317 21:19:06.265313 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet cef932d50d4544ad9c5ca245bc006f29
I20260317 21:19:06.270722 7010 tablet_service.cc:1558] Processing DeleteTablet for tablet cef932d50d4544ad9c5ca245bc006f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:06.279793 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f95f849f2a4340bbb5f515633e6f50fe" } modify_external_catalogs: true
I20260317 21:19:06.280365 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f95f849f2a4340bbb5f515633e6f50fe" } modify_external_catalogs: true
I20260317 21:19:06.282387 7038 raft_consensus.cc:2468] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:06.285063 6674 catalog_manager.cc:5671] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:06.286592 6674 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
I20260317 21:19:06.285921 7995 raft_consensus.cc:359] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 1 FOLLOWER]: Replica starting. Triggering 4 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:06.301561 7869 log.cc:1199] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/2562cc2a3bb2430981c88475f9e1daa1
I20260317 21:19:06.302469 7869 ts_tablet_manager.cc:1950] T 2562cc2a3bb2430981c88475f9e1daa1 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:06.273897 7152 tablet_service.cc:1558] Processing DeleteTablet for tablet cef932d50d4544ad9c5ca245bc006f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:06.224102 7467 consensus_queue.cc:260] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.303668 7181 raft_consensus.cc:2468] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:06.307300 7467 ts_tablet_manager.cc:1434] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.154s user 0.005s sys 0.000s
I20260317 21:19:06.309871 7120 leader_election.cc:304] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:06.311480 8008 raft_consensus.cc:2804] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:06.311945 8008 raft_consensus.cc:697] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:06.312928 7462 ts_tablet_manager.cc:1403] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.279s user 0.006s sys 0.009s
I20260317 21:19:06.313236 7467 tablet_bootstrap.cc:492] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:06.312763 8008 consensus_queue.cc:237] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.315765 7462 raft_consensus.cc:359] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.291147 7459 tablet_bootstrap.cc:492] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:06.316406 7462 raft_consensus.cc:385] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.305514 6971 leader_election.cc:304] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:06.316699 7462 raft_consensus.cc:740] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.316985 7459 ts_tablet_manager.cc:1403] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.269s user 0.014s sys 0.000s
I20260317 21:19:06.317423 8028 raft_consensus.cc:2804] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:06.317477 7995 raft_consensus.cc:740] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47e1c983d0224428a9a7b147a0e5e4ef, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.317304 7462 consensus_queue.cc:260] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.317777 8028 raft_consensus.cc:493] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:06.318090 8028 raft_consensus.cc:3060] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:06.319051 7995 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:06.327812 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet bcd99ca9f6c540e48e29d8659fbfe777 (table default.Table_fcec6e585d6245ff828c8c0c98450d63 [id=d238f6cb466c49d6af5ae380e967b054]) successfully deleted
I20260317 21:19:06.327858 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet cef932d50d4544ad9c5ca245bc006f29 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:06.319702 7459 raft_consensus.cc:359] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.336885 7459 raft_consensus.cc:385] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.346843 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:06.347676 7459 raft_consensus.cc:740] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.348839 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 208) took 2797 ms. Trace:
I20260317 21:19:06.349300 7869 rpcz_store.cc:276] 0317 21:19:03.550952 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.560759 (+ 9807us) service_pool.cc:225] Handling call
0317 21:19:05.570976 (+2010217us) ts_tablet_manager.cc:1151] Deleting tablet 2562cc2a3bb2430981c88475f9e1daa1
0317 21:19:05.577758 (+ 6782us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.042624 (+464866us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:06.244043 (+201419us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:06.348773 (+104730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2009535}
I20260317 21:19:06.351475 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 2562cc2a3bb2430981c88475f9e1daa1 (table default.Table_0b009e9d95134391a1d2d41e80db86c7 [id=6d460b03e35a454da7657c608a0f87cf]) successfully deleted
I20260317 21:19:06.353447 7459 consensus_queue.cc:260] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.357774 8028 raft_consensus.cc:515] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:06.363802 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet cef932d50d4544ad9c5ca245bc006f29 because the tablet was not found. No further retry: Not found: Tablet not found: cef932d50d4544ad9c5ca245bc006f29
I20260317 21:19:06.373621 8008 consensus_queue.cc:1048] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:06.371448 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 210) took 2601 ms. Trace:
I20260317 21:19:06.392671 7459 ts_tablet_manager.cc:1434] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.075s user 0.006s sys 0.000s
I20260317 21:19:06.415712 7869 rpcz_store.cc:276] 0317 21:19:03.752252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:03.763162 (+ 10910us) service_pool.cc:225] Handling call
0317 21:19:06.352533 (+2589371us) ts_tablet_manager.cc:1151] Deleting tablet c9b7f7c864004b9abde0a0201e196165
0317 21:19:06.352611 (+ 78us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.353117 (+ 506us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2588571}
I20260317 21:19:06.417160 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 214) took 2294 ms. Trace:
I20260317 21:19:06.417536 7869 rpcz_store.cc:276] 0317 21:19:04.122972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.123107 (+ 135us) service_pool.cc:225] Handling call
0317 21:19:06.416799 (+2293692us) ts_tablet_manager.cc:1151] Deleting tablet 16f64073188f4d06859e1a701df73242
0317 21:19:06.416845 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.417108 (+ 263us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2293075}
I20260317 21:19:06.418799 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 216) took 2021 ms. Trace:
I20260317 21:19:06.420327 8028 leader_election.cc:290] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:06.424038 6670 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a4d64c48641b484ab443380808544d2f consensus state has no leader
I20260317 21:19:06.426614 7467 tablet_bootstrap.cc:654] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.427264 8028 raft_consensus.cc:493] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:06.428297 8028 raft_consensus.cc:515] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:06.429603 6973 leader_election.cc:343] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 2b287c83b77548f19546927b05905746
W20260317 21:19:06.431418 6672 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 5e080308f93448138a4846ab6bec7dfa has no consensus state
W20260317 21:19:06.433465 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet c9b7f7c864004b9abde0a0201e196165 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c9b7f7c864004b9abde0a0201e196165 already in progress: creating tablet
W20260317 21:19:06.434165 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 16f64073188f4d06859e1a701df73242 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 16f64073188f4d06859e1a701df73242 already in progress: creating tablet
I20260317 21:19:06.439467 6667 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:06.439002 7462 ts_tablet_manager.cc:1434] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.126s user 0.006s sys 0.001s
I20260317 21:19:06.442941 7462 tablet_bootstrap.cc:492] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:06.450095 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:06.451151 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:06.452543 7995 sys_catalog.cc:565] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:19:06.455827 7459 tablet_bootstrap.cc:492] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:06.432464 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2b287c83b77548f19546927b05905746" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:06.435590 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78e2e20d320547b3bb1e03923de52822" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:06.450495 7869 rpcz_store.cc:276] 0317 21:19:04.396949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.397083 (+ 134us) service_pool.cc:225] Handling call
0317 21:19:06.418458 (+2021375us) ts_tablet_manager.cc:1151] Deleting tablet 1c3beb9232d74e45ab713ae8241eec4c
0317 21:19:06.418495 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.418749 (+ 254us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1977372}
W20260317 21:19:06.487656 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 1c3beb9232d74e45ab713ae8241eec4c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c3beb9232d74e45ab713ae8241eec4c already in progress: creating tablet
I20260317 21:19:06.491845 8028 leader_election.cc:290] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:06.493402 6973 leader_election.cc:343] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 78e2e20d320547b3bb1e03923de52822
W20260317 21:19:06.513361 7119 consensus_peers.cc:597] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0eee8b5588c24661889c593797c94269. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:06.516175 6971 leader_election.cc:343] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.516634 6971 leader_election.cc:304] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:06.518090 8028 raft_consensus.cc:2749] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:06.515713 7181 raft_consensus.cc:3060] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:06.521293 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:06.525111 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet b9d1076839894509805be59b751aa847
I20260317 21:19:06.526502 6671 catalog_manager.cc:6741] ReplaceTablet: tablet b9d1076839894509805be59b751aa847 deleted and replaced by tablet d2aba7eba6df430e86e003ef3a179912
I20260317 21:19:06.528294 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet b9d1076839894509805be59b751aa847 with delete_type TABLET_DATA_DELETED (replaced by tablet d2aba7eba6df430e86e003ef3a179912) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:06.531492 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet b9d1076839894509805be59b751aa847 with delete_type TABLET_DATA_DELETED (replaced by tablet d2aba7eba6df430e86e003ef3a179912) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:06.531579 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet b9d1076839894509805be59b751aa847 with delete_type TABLET_DATA_DELETED (replaced by tablet d2aba7eba6df430e86e003ef3a179912) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:06.535321 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet b9d1076839894509805be59b751aa847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9d1076839894509805be59b751aa847 already in progress: creating tablet
W20260317 21:19:06.540426 7119 consensus_peers.cc:597] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: eb332f981fc84a178fadae2f4908bdde. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:06.548524 7869 tablet_replica.cc:333] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:06.549373 7869 raft_consensus.cc:2243] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:06.550321 7869 pending_rounds.cc:70] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:19:06.550580 7869 pending_rounds.cc:77] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412486777806848 op_type: NO_OP noop_request { }
I20260317 21:19:06.550925 7869 raft_consensus.cc:2889] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:06.551244 7869 raft_consensus.cc:2272] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:06.553004 8028 raft_consensus.cc:493] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:06.553428 7869 ts_tablet_manager.cc:1916] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:06.553495 8028 raft_consensus.cc:515] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.555960 8028 leader_election.cc:290] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:06.556941 7181 raft_consensus.cc:2468] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:06.557847 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:06.558432 7328 raft_consensus.cc:2468] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:06.558578 6671 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0d5bd4305663413496306f96a3b22926" } modify_external_catalogs: true
I20260317 21:19:06.559190 6671 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_0d5bd4305663413496306f96a3b22926" } modify_external_catalogs: true
I20260317 21:19:06.560959 6973 leader_election.cc:304] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:06.562387 8028 raft_consensus.cc:2804] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:06.562875 8028 raft_consensus.cc:493] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:06.563730 8028 raft_consensus.cc:3060] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:06.565086 6971 leader_election.cc:304] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:06.566744 7994 raft_consensus.cc:2804] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:06.575906 7462 tablet_bootstrap.cc:654] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.583292 7918 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:06.587098 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:06.588372 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet d4afbb1ab93a44d9ab5795b72444c690
I20260317 21:19:06.591917 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet d4afbb1ab93a44d9ab5795b72444c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:06.592604 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet d4afbb1ab93a44d9ab5795b72444c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:06.593878 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet d4afbb1ab93a44d9ab5795b72444c690 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:06.595346 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet d4afbb1ab93a44d9ab5795b72444c690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4afbb1ab93a44d9ab5795b72444c690 already in progress: creating tablet
I20260317 21:19:06.599182 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:06.606760 6971 leader_election.cc:343] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.584415 8028 raft_consensus.cc:515] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.616431 7467 tablet_bootstrap.cc:492] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:06.616977 7467 ts_tablet_manager.cc:1403] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.304s user 0.013s sys 0.007s
I20260317 21:19:06.623389 7467 raft_consensus.cc:359] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.624109 7467 raft_consensus.cc:385] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.624359 7467 raft_consensus.cc:740] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.625916 7462 tablet_bootstrap.cc:492] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:06.626389 7462 ts_tablet_manager.cc:1403] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.184s user 0.009s sys 0.002s
I20260317 21:19:06.629587 6674 master_service.cc:954] ReplaceTablet: received request to replace tablet 5b066a6a5c864beb8da59bd97bacd309 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:06.633518 7462 raft_consensus.cc:359] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.637112 7994 raft_consensus.cc:697] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:06.637264 7467 consensus_queue.cc:260] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.625036 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:06.645697 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_20aa87dfa03e46f5b2db253720a9e012" } new_table_name: "default.Table_f9ab9898a12b4ffeb9a2c721ee8f05c3" modify_external_catalogs: true
W20260317 21:19:06.646790 6971 leader_election.cc:343] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.648128 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_6bb2d165f08a40d7bd84414364358f7e" } new_table_name: "default.Table_24ec4c242298468dba3b539b6f272362" modify_external_catalogs: true
I20260317 21:19:06.653771 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:06.653111 7467 ts_tablet_manager.cc:1434] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.032s user 0.006s sys 0.000s
I20260317 21:19:06.659608 7994 consensus_queue.cc:237] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.662830 7869 ts_tablet_manager.cc:1929] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:06.663328 7869 log.cc:1199] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/eb332f981fc84a178fadae2f4908bdde
I20260317 21:19:06.664204 7869 ts_tablet_manager.cc:1950] T eb332f981fc84a178fadae2f4908bdde P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:06.645074 8028 leader_election.cc:290] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:06.664592 6639 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 48872.9us
I20260317 21:19:06.639976 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:06.667505 7462 raft_consensus.cc:385] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.667637 7328 raft_consensus.cc:3060] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:06.667835 7462 raft_consensus.cc:740] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.668493 7462 consensus_queue.cc:260] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.673969 8038 raft_consensus.cc:493] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:06.674597 8038 raft_consensus.cc:515] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.676517 8038 leader_election.cc:290] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:06.678107 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:06.679497 7264 leader_election.cc:343] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.682509 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:06.685539 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 829bc3959a9e4c2dbb6277cde5969b0e (table default.Table_4e0de322dbf141f38753722166148d81 [id=2652ee9fdb984f14a138c570bda4636d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 829bc3959a9e4c2dbb6277cde5969b0e consensus state has no leader
W20260317 21:19:06.685993 7261 leader_election.cc:343] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:06.686936 7261 leader_election.cc:304] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:06.688665 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 32be4818a67a41c998f04d7cedf7356e has no consensus state
I20260317 21:19:06.689350 8038 raft_consensus.cc:2749] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:06.690155 7459 tablet_bootstrap.cc:654] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:06.689594 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b8c9cd4f760f44c88cdd986542755443 consensus state has no leader
I20260317 21:19:06.700380 8028 consensus_queue.cc:1048] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:06.701823 7467 tablet_bootstrap.cc:492] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:06.702570 7328 raft_consensus.cc:2468] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:06.704165 6973 leader_election.cc:304] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:06.705394 8060 raft_consensus.cc:2804] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:06.707782 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_cab5f9b577f542c7a07d30cfd4c66fdb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:06.709586 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_32379e31919e4013961f12a0c3692b4b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:06.710255 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cab5f9b577f542c7a07d30cfd4c66fdb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:06.711136 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3a3ba3d4d60c4441a9f75ff76812f254"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:06.711483 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32379e31919e4013961f12a0c3692b4b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:06.712803 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a3ba3d4d60c4441a9f75ff76812f254 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:06.714516 7791 ts_tablet_manager.cc:1929] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:06.714785 8060 raft_consensus.cc:697] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:06.715080 7791 log.cc:1199] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/eb332f981fc84a178fadae2f4908bdde
I20260317 21:19:06.718936 7462 ts_tablet_manager.cc:1434] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.092s user 0.005s sys 0.000s
I20260317 21:19:06.720777 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 217) took 2270 ms. Trace:
I20260317 21:19:06.721982 7462 tablet_bootstrap.cc:492] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:06.729233 7467 tablet_bootstrap.cc:654] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.721333 7869 rpcz_store.cc:276] 0317 21:19:04.450340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.450608 (+ 268us) service_pool.cc:225] Handling call
0317 21:19:06.487057 (+2036449us) ts_tablet_manager.cc:1151] Deleting tablet eb332f981fc84a178fadae2f4908bdde
0317 21:19:06.548398 (+ 61341us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.611611 (+ 63213us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:06.650345 (+ 38734us) spinlock_profiling.cc:243] Waited 12 ms on lock 0x7b5c000d238c. stack: 00007f622fe56423 00007f622fe56320 00007f622f9cb574 0000000000509544 00000000005094d9 00007f62372e9951 00007f6237487875 00007f6237483f56 00007f6237484f87 00007f6237fc7038 00007f6237fc9970 00007f6237ffa510 00007f6237fe0add 00007f6237fe0a89 00007f6237fe0a19 00007f6237fe09e1
0317 21:19:06.662601 (+ 12256us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:06.720688 (+ 58087us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":27518720,"tablet-delete.queue_time_us":2035777}
I20260317 21:19:06.731979 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]) successfully deleted
I20260317 21:19:06.733920 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 219) took 2260 ms. Trace:
I20260317 21:19:06.734454 7869 rpcz_store.cc:276] 0317 21:19:04.473098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.473215 (+ 117us) service_pool.cc:225] Handling call
0317 21:19:06.733455 (+2260240us) ts_tablet_manager.cc:1151] Deleting tablet 2607b9978c32466eb6dc8b8acfb429ce
0317 21:19:06.733513 (+ 58us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.733846 (+ 333us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2253386}
W20260317 21:19:06.738418 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 2607b9978c32466eb6dc8b8acfb429ce because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2607b9978c32466eb6dc8b8acfb429ce already in progress: creating tablet
I20260317 21:19:06.739652 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 220) took 2009 ms. Trace:
I20260317 21:19:06.740046 7869 rpcz_store.cc:276] 0317 21:19:04.730257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.730361 (+ 104us) service_pool.cc:225] Handling call
0317 21:19:06.739291 (+2008930us) ts_tablet_manager.cc:1151] Deleting tablet 6ab5f3dc7836425d917bd09f1e6bfe90
0317 21:19:06.739334 (+ 43us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.739593 (+ 259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2008366}
W20260317 21:19:06.741498 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ab5f3dc7836425d917bd09f1e6bfe90 already in progress: creating tablet
I20260317 21:19:06.742622 7869 tablet_replica.cc:333] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:06.743398 7869 raft_consensus.cc:2243] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:06.750329 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_8d558e5446ca41ec87974c9e1cec387e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:06.751884 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8d558e5446ca41ec87974c9e1cec387e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:06.755148 7462 tablet_bootstrap.cc:654] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.763609 7918 pending_rounds.cc:85] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Aborting all ops after (but not including) 5
I20260317 21:19:06.764201 7918 pending_rounds.cc:107] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Aborting uncommitted WRITE_OP operation due to leader change: 1.6
I20260317 21:19:06.766942 7918 pending_rounds.cc:107] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Aborting uncommitted WRITE_OP operation due to leader change: 1.7
I20260317 21:19:06.748541 7869 raft_consensus.cc:2272] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:06.768440 7467 tablet_bootstrap.cc:492] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:06.768919 7467 ts_tablet_manager.cc:1403] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.067s user 0.019s sys 0.002s
I20260317 21:19:06.771522 7467 raft_consensus.cc:359] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.772208 7467 raft_consensus.cc:385] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.772455 7467 raft_consensus.cc:740] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.774708 7869 ts_tablet_manager.cc:1916] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:06.780164 7462 tablet_bootstrap.cc:492] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:06.780822 7462 ts_tablet_manager.cc:1403] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.061s user 0.010s sys 0.001s
I20260317 21:19:06.781896 7918 pending_rounds.cc:107] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Aborting uncommitted WRITE_OP operation due to leader change: 1.8
I20260317 21:19:06.783730 7462 raft_consensus.cc:359] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.784376 7462 raft_consensus.cc:385] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.784724 7462 raft_consensus.cc:740] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.785383 7462 consensus_queue.cc:260] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.716317 7791 ts_tablet_manager.cc:1950] T eb332f981fc84a178fadae2f4908bdde P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:06.800568 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 212) took 2350 ms. Trace:
I20260317 21:19:06.717016 8060 consensus_queue.cc:237] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.801040 7791 rpcz_store.cc:276] 0317 21:19:04.449888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.511712 (+ 61824us) service_pool.cc:225] Handling call
0317 21:19:06.104601 (+1592889us) ts_tablet_manager.cc:1151] Deleting tablet eb332f981fc84a178fadae2f4908bdde
0317 21:19:06.104702 (+ 101us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.452279 (+347577us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:06.714150 (+261871us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:06.800493 (+ 86343us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":37572,"tablet-delete.queue_time_us":1591129}
I20260317 21:19:06.802991 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 215) took 2272 ms. Trace:
I20260317 21:19:06.803056 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet eb332f981fc84a178fadae2f4908bdde (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]) successfully deleted
I20260317 21:19:06.803295 7791 rpcz_store.cc:276] 0317 21:19:04.530205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.531835 (+ 1630us) service_pool.cc:225] Handling call
0317 21:19:06.802639 (+2270804us) ts_tablet_manager.cc:1151] Deleting tablet 6ab5f3dc7836425d917bd09f1e6bfe90
0317 21:19:06.802683 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.802939 (+ 256us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2270168}
I20260317 21:19:06.804291 7791 tablet_replica.cc:333] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:06.806077 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 6ab5f3dc7836425d917bd09f1e6bfe90 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ab5f3dc7836425d917bd09f1e6bfe90 already in progress: creating tablet
I20260317 21:19:06.813107 6666 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e717b96ed78c4b5d8aab97d49666ebea" } modify_external_catalogs: true
I20260317 21:19:06.814040 6666 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e717b96ed78c4b5d8aab97d49666ebea" } modify_external_catalogs: true
I20260317 21:19:06.826848 8028 consensus_queue.cc:1048] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:06.814602 7120 consensus_peers.cc:597] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:06.851424 7459 tablet_bootstrap.cc:492] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:06.851863 7459 ts_tablet_manager.cc:1403] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.396s user 0.007s sys 0.007s
I20260317 21:19:06.855077 7459 raft_consensus.cc:359] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.855752 7459 raft_consensus.cc:385] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:06.856011 7459 raft_consensus.cc:740] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:06.856590 7459 consensus_queue.cc:260] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.871723 7459 ts_tablet_manager.cc:1434] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.020s user 0.004s sys 0.003s
I20260317 21:19:06.866075 7791 raft_consensus.cc:2243] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:06.873592 7791 raft_consensus.cc:2272] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:06.875506 7791 ts_tablet_manager.cc:1916] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:06.875931 8076 tablet.cc:1723] T e4522181cc434203ac14c9b07587fec8 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:19:06.878402 8077 tablet.cc:1723] T e4522181cc434203ac14c9b07587fec8 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:19:06.869860 7462 ts_tablet_manager.cc:1434] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.089s user 0.006s sys 0.000s
I20260317 21:19:06.883090 7462 tablet_bootstrap.cc:492] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:06.813668 7467 consensus_queue.cc:260] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:06.890519 7467 ts_tablet_manager.cc:1434] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.121s user 0.005s sys 0.000s
I20260317 21:19:06.894656 6667 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_66c25b9b83724e718f148f260ec5f672" } new_table_name: "default.Table_f9013931d28a41148cb95eda076837f7" modify_external_catalogs: true
I20260317 21:19:06.897094 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_76a974b216ba482e9783ae7f126c5d31" } new_table_name: "default.Table_1f45d1a0799640caab2984b6590f965e" modify_external_catalogs: true
I20260317 21:19:06.905215 8075 tablet.cc:1723] T e4522181cc434203ac14c9b07587fec8 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:19:06.915836 7459 tablet_bootstrap.cc:492] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:06.931054 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8a44a297079347bab6f8be6abe332bfa consensus state has no leader
I20260317 21:19:06.891559 7467 tablet_bootstrap.cc:492] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:06.957095 7459 tablet_bootstrap.cc:654] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:06.976864 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:07.000659 7083 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:07.025439 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_66613ac613e64bf6a8f6afd413d96c34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.026422 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_dfe3f4e660b042fca5a8ce85c8816ad4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.027427 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1cc7099df8b34e2ba85b14761993b230"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.037076 7791 ts_tablet_manager.cc:1929] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260317 21:19:07.037464 7791 log.cc:1199] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/f749358db0f74ef78d9eee23b72bfe3e
I20260317 21:19:07.038164 7791 ts_tablet_manager.cc:1950] T f749358db0f74ef78d9eee23b72bfe3e P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.042505 7462 tablet_bootstrap.cc:654] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.045923 7869 ts_tablet_manager.cc:1929] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:07.046411 7869 log.cc:1199] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/0eee8b5588c24661889c593797c94269
I20260317 21:19:07.047228 7869 ts_tablet_manager.cc:1950] T 0eee8b5588c24661889c593797c94269 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
W20260317 21:19:07.048111 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1cc7099df8b34e2ba85b14761993b230 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:07.048904 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_66613ac613e64bf6a8f6afd413d96c34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.059653 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 219) took 2267 ms. Trace:
I20260317 21:19:07.060022 7791 rpcz_store.cc:276] 0317 21:19:04.792067 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.792181 (+ 114us) service_pool.cc:225] Handling call
0317 21:19:06.804099 (+2011918us) ts_tablet_manager.cc:1151] Deleting tablet f749358db0f74ef78d9eee23b72bfe3e
0317 21:19:06.804151 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.973226 (+169075us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.036847 (+ 63621us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.059587 (+ 22740us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2002915}
I20260317 21:19:07.061369 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]) successfully deleted
I20260317 21:19:07.062297 7791 tablet_replica.cc:333] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:07.062981 7791 raft_consensus.cc:2243] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.063443 7791 raft_consensus.cc:2272] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.066329 7791 ts_tablet_manager.cc:1916] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.066464 7459 tablet_bootstrap.cc:492] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:07.067296 7459 ts_tablet_manager.cc:1403] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.195s user 0.023s sys 0.002s
I20260317 21:19:07.070262 7459 raft_consensus.cc:359] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.071154 7459 raft_consensus.cc:385] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.071470 7459 raft_consensus.cc:740] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.072230 7459 consensus_queue.cc:260] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.082237 7459 ts_tablet_manager.cc:1434] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.015s user 0.006s sys 0.000s
I20260317 21:19:07.083312 7459 tablet_bootstrap.cc:492] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:07.101913 8060 raft_consensus.cc:493] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.102406 8060 raft_consensus.cc:515] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.105808 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:07.109385 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:07.110153 7326 raft_consensus.cc:2468] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
W20260317 21:19:07.111845 6971 leader_election.cc:343] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.112916 6973 leader_election.cc:304] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:07.113721 8028 raft_consensus.cc:2804] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:07.114050 8028 raft_consensus.cc:493] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:07.114348 8028 raft_consensus.cc:3060] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.124989 7937 raft_consensus.cc:493] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.125453 7937 raft_consensus.cc:515] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.129932 7462 tablet_bootstrap.cc:492] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:07.130336 7462 ts_tablet_manager.cc:1403] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.247s user 0.009s sys 0.002s
W20260317 21:19:07.130225 7119 leader_election.cc:343] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 78e2e20d320547b3bb1e03923de52822
I20260317 21:19:07.131703 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78e2e20d320547b3bb1e03923de52822" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.132295 7036 raft_consensus.cc:2468] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:07.135951 8038 raft_consensus.cc:493] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.136417 8038 raft_consensus.cc:515] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.138482 8038 leader_election.cc:290] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:07.139006 7459 tablet_bootstrap.cc:654] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.142434 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.104530 8060 leader_election.cc:290] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:07.145666 7264 leader_election.cc:343] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.149291 6669 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_687dd24cc64a425b80bcfb73ef08d1fd" } new_table_name: "default.Table_d6e9cf5141a346f7af807b9f5105eb52" modify_external_catalogs: true
I20260317 21:19:07.152998 7120 leader_election.cc:304] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
W20260317 21:19:07.154070 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.156980 8008 raft_consensus.cc:2804] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:07.157354 8008 raft_consensus.cc:493] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:07.157641 8008 raft_consensus.cc:3060] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.128264 7937 leader_election.cc:290] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.162238 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:07.163839 7261 leader_election.cc:343] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.164214 7261 leader_election.cc:304] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:07.165475 8038 raft_consensus.cc:2749] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:07.166785 7459 tablet_bootstrap.cc:492] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:07.167281 7459 ts_tablet_manager.cc:1403] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.003s
I20260317 21:19:07.170423 7459 raft_consensus.cc:359] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.171296 7459 raft_consensus.cc:385] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.171622 7459 raft_consensus.cc:740] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.178350 7467 tablet_bootstrap.cc:654] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.132519 7462 raft_consensus.cc:359] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.181074 7462 raft_consensus.cc:385] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.181342 7462 raft_consensus.cc:740] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.181530 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_b945e0f5a11c4ca0b4a05171094fbfe5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.182317 7462 consensus_queue.cc:260] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:07.183357 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.172367 7459 consensus_queue.cc:260] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.191980 8028 raft_consensus.cc:515] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.195179 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:07.196566 6971 leader_election.cc:343] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.208686 7459 ts_tablet_manager.cc:1434] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.020s user 0.008s sys 0.000s
I20260317 21:19:07.210000 7459 tablet_bootstrap.cc:492] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:07.216125 7462 ts_tablet_manager.cc:1434] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.086s user 0.006s sys 0.000s
I20260317 21:19:07.237104 7374 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:07.237645 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:07.239531 7459 tablet_bootstrap.cc:654] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.240363 7462 tablet_bootstrap.cc:492] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:07.261722 6673 master_service.cc:954] ReplaceTablet: received request to replace tablet 275e9cd7f14040d98eb85a6bb1010c2c from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:07.267858 7937 consensus_queue.cc:1048] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:07.175185 8008 raft_consensus.cc:515] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.272838 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "78e2e20d320547b3bb1e03923de52822" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:07.273253 8008 leader_election.cc:290] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.273699 7038 raft_consensus.cc:3060] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.274683 7459 tablet_bootstrap.cc:492] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:07.275070 7459 ts_tablet_manager.cc:1403] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.065s user 0.008s sys 0.004s
W20260317 21:19:07.275166 7119 leader_election.cc:343] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 78e2e20d320547b3bb1e03923de52822
W20260317 21:19:07.277423 7119 consensus_peers.cc:597] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:07.277448 7459 raft_consensus.cc:359] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.278131 7459 raft_consensus.cc:385] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.278762 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 223) took 2542 ms. Trace:
I20260317 21:19:07.278399 7459 raft_consensus.cc:740] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:07.278445 6669 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0dd4c1efc0624957832c17dc7a4db3f3 consensus state has no leader
I20260317 21:19:07.279086 7869 rpcz_store.cc:276] 0317 21:19:04.736688 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:04.736794 (+ 106us) service_pool.cc:225] Handling call
0317 21:19:06.742414 (+2005620us) ts_tablet_manager.cc:1151] Deleting tablet 0eee8b5588c24661889c593797c94269
0317 21:19:06.742465 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:06.835490 (+ 93025us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.045647 (+210157us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.278701 (+233054us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2004990}
I20260317 21:19:07.279381 7459 consensus_queue.cc:260] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.280139 7869 tablet_replica.cc:333] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:07.281335 7869 raft_consensus.cc:2243] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:07.282332 7869 raft_consensus.cc:2272] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.284276 7869 ts_tablet_manager.cc:1916] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.284400 7038 raft_consensus.cc:2468] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:07.285221 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 0eee8b5588c24661889c593797c94269 (table default.Table_212ed8405de344519eaecc32555fcbb5 [id=8fee88b3acf649ffbfbc18d16f47af40]) successfully deleted
I20260317 21:19:07.286430 7120 leader_election.cc:304] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:07.287866 7937 raft_consensus.cc:2804] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:07.294493 7937 raft_consensus.cc:697] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:07.295617 6669 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_dc3c4bce28a841739bb2887cf3a40e15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.297111 7937 consensus_queue.cc:237] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:07.297946 6669 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc3c4bce28a841739bb2887cf3a40e15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.298640 7459 ts_tablet_manager.cc:1434] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.023s user 0.006s sys 0.001s
I20260317 21:19:07.301698 8038 raft_consensus.cc:493] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.302178 8038 raft_consensus.cc:515] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.305783 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:07.306658 8038 leader_election.cc:290] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:07.307754 7261 leader_election.cc:343] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.317871 8028 leader_election.cc:290] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.318562 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:07.319156 7326 raft_consensus.cc:3060] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.322409 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:07.323674 7264 leader_election.cc:343] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.324211 7459 tablet_bootstrap.cc:492] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:07.324093 7264 leader_election.cc:304] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:07.328233 8038 raft_consensus.cc:2749] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:07.327857 7462 tablet_bootstrap.cc:654] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.333011 7326 raft_consensus.cc:2468] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:07.351817 6973 leader_election.cc:304] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:07.360421 7791 ts_tablet_manager.cc:1929] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:07.360883 7791 log.cc:1199] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/0eee8b5588c24661889c593797c94269
I20260317 21:19:07.364138 7467 tablet_bootstrap.cc:492] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:07.368510 8028 raft_consensus.cc:2804] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:07.369436 8028 raft_consensus.cc:697] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:07.375211 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_846f03c95b0340468fae274b89fcf350"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.364763 7467 ts_tablet_manager.cc:1403] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.473s user 0.011s sys 0.005s
I20260317 21:19:07.376542 8028 consensus_queue.cc:237] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:07.392061 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_846f03c95b0340468fae274b89fcf350 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.406282 7791 ts_tablet_manager.cc:1950] T 0eee8b5588c24661889c593797c94269 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.407172 7467 raft_consensus.cc:359] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.407894 7467 raft_consensus.cc:385] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.408169 7467 raft_consensus.cc:740] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.409060 7467 consensus_queue.cc:260] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.410360 7459 tablet_bootstrap.cc:654] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.430442 7467 ts_tablet_manager.cc:1434] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.053s user 0.005s sys 0.000s
I20260317 21:19:07.443408 7467 tablet_bootstrap.cc:492] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:07.450379 7459 tablet_bootstrap.cc:492] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:07.450951 7459 ts_tablet_manager.cc:1403] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.127s user 0.012s sys 0.000s
I20260317 21:19:07.453404 7459 raft_consensus.cc:359] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.454257 7459 raft_consensus.cc:385] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.454597 7459 raft_consensus.cc:740] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.455266 7459 consensus_queue.cc:260] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.457371 7459 ts_tablet_manager.cc:1434] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260317 21:19:07.477203 8008 raft_consensus.cc:493] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.477895 8008 raft_consensus.cc:515] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.496315 8038 raft_consensus.cc:493] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.496868 8038 raft_consensus.cc:515] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.502830 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:07.503644 7181 raft_consensus.cc:2468] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:07.506073 7261 leader_election.cc:304] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:07.508052 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:07.508976 7326 raft_consensus.cc:2468] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:07.510731 7119 leader_election.cc:304] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:07.511725 8038 leader_election.cc:290] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:07.512528 8038 raft_consensus.cc:2804] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:07.513010 8038 raft_consensus.cc:493] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:07.516690 7937 raft_consensus.cc:2804] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:07.513758 8038 raft_consensus.cc:3060] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.517163 7937 raft_consensus.cc:493] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:07.517567 7937 raft_consensus.cc:3060] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.526746 7869 ts_tablet_manager.cc:1929] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260317 21:19:07.527441 7869 log.cc:1199] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/f749358db0f74ef78d9eee23b72bfe3e
I20260317 21:19:07.528542 7869 ts_tablet_manager.cc:1950] T f749358db0f74ef78d9eee23b72bfe3e P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:07.531699 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 225) took 2517 ms. Trace:
I20260317 21:19:07.538071 8038 raft_consensus.cc:515] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.550922 7462 tablet_bootstrap.cc:492] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:07.551499 7462 ts_tablet_manager.cc:1403] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.311s user 0.007s sys 0.004s
I20260317 21:19:07.553083 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.553304 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:07.553681 7036 raft_consensus.cc:2468] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:07.557258 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.557821 7038 raft_consensus.cc:2468] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:07.559705 8038 leader_election.cc:290] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:07.571192 7459 tablet_bootstrap.cc:492] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:07.577838 7459 tablet_bootstrap.cc:654] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.577996 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 223) took 2576 ms. Trace:
I20260317 21:19:07.578399 7791 rpcz_store.cc:276] 0317 21:19:05.001571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.002300 (+ 729us) service_pool.cc:225] Handling call
0317 21:19:07.062121 (+2059821us) ts_tablet_manager.cc:1151] Deleting tablet 0eee8b5588c24661889c593797c94269
0317 21:19:07.062177 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.164746 (+102569us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.360165 (+195419us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.577934 (+217769us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2059148}
I20260317 21:19:07.579699 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 0eee8b5588c24661889c593797c94269 (table default.Table_212ed8405de344519eaecc32555fcbb5 [id=8fee88b3acf649ffbfbc18d16f47af40]) successfully deleted
I20260317 21:19:07.579983 7791 tablet_replica.cc:333] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:07.580837 7791 raft_consensus.cc:2243] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.581296 7791 raft_consensus.cc:2272] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.583779 7791 ts_tablet_manager.cc:1916] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.507098 8008 leader_election.cc:290] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.585700 7869 rpcz_store.cc:276] 0317 21:19:05.013870 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.013966 (+ 96us) service_pool.cc:225] Handling call
0317 21:19:07.279993 (+2266027us) ts_tablet_manager.cc:1151] Deleting tablet f749358db0f74ef78d9eee23b72bfe3e
0317 21:19:07.280033 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.395702 (+115669us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.526396 (+130694us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.531623 (+ 5227us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2265512}
I20260317 21:19:07.587272 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet f749358db0f74ef78d9eee23b72bfe3e (table default.Table_411f7f4c71494d26b681f5e76fa36214 [id=944a00d9ea784873b8e346589b9d86b6]) successfully deleted
I20260317 21:19:07.587232 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 232) took 2149 ms. Trace:
I20260317 21:19:07.536056 7467 tablet_bootstrap.cc:654] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.557883 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:07.590243 8038 raft_consensus.cc:493] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.590839 8038 raft_consensus.cc:515] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.591845 7459 tablet_bootstrap.cc:492] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:07.592301 7459 ts_tablet_manager.cc:1403] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.021s user 0.014s sys 0.004s
I20260317 21:19:07.594426 8038 leader_election.cc:290] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:07.594409 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40549347f8463797d6791136883371" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.595192 7459 raft_consensus.cc:359] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:07.595695 7264 leader_election.cc:343] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.595816 7459 raft_consensus.cc:385] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.596055 7459 raft_consensus.cc:740] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.595989 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40549347f8463797d6791136883371" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:07.596911 7459 consensus_queue.cc:260] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:07.597302 7261 leader_election.cc:343] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.597673 7261 leader_election.cc:304] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:07.598786 8038 raft_consensus.cc:2749] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:07.599327 7459 ts_tablet_manager.cc:1434] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:07.600453 7459 tablet_bootstrap.cc:492] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:07.601192 7462 raft_consensus.cc:359] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.601815 7462 raft_consensus.cc:385] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.602404 7462 raft_consensus.cc:740] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.608356 7459 tablet_bootstrap.cc:654] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.595412 7937 raft_consensus.cc:515] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.612936 7181 raft_consensus.cc:2393] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f539d1a4a584b8ca83928be49369bf4 in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:19:07.612269 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_36d9fab3efe44666a2ce8cde9311325d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:07.616222 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36d9fab3efe44666a2ce8cde9311325d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.618961 7937 leader_election.cc:290] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.619380 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:07.620002 7038 raft_consensus.cc:3060] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.587996 7869 rpcz_store.cc:276] 0317 21:19:05.437395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.437513 (+ 118us) service_pool.cc:225] Handling call
0317 21:19:07.586846 (+2149333us) ts_tablet_manager.cc:1151] Deleting tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e
0317 21:19:07.586894 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.587174 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2148774}
I20260317 21:19:07.621685 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 235) took 1975 ms. Trace:
I20260317 21:19:07.621982 7869 rpcz_store.cc:276] 0317 21:19:05.646056 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.646169 (+ 113us) service_pool.cc:225] Handling call
0317 21:19:07.621321 (+1975152us) ts_tablet_manager.cc:1151] Deleting tablet 5314c562bc9d457a8e1f4930e8d3a7ff
0317 21:19:07.621362 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.621630 (+ 268us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1921375}
I20260317 21:19:07.623412 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 239) took 1855 ms. Trace:
I20260317 21:19:07.623704 7869 rpcz_store.cc:276] 0317 21:19:05.768379 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.768516 (+ 137us) service_pool.cc:225] Handling call
0317 21:19:07.623091 (+1854575us) ts_tablet_manager.cc:1151] Deleting tablet ae1b2a549af6485ca3faee3f0ad845a2
0317 21:19:07.623129 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.623364 (+ 235us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1853942}
I20260317 21:19:07.624775 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 240) took 1786 ms. Trace:
I20260317 21:19:07.625042 7869 rpcz_store.cc:276] 0317 21:19:05.838170 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.839059 (+ 889us) service_pool.cc:225] Handling call
0317 21:19:07.624413 (+1785354us) ts_tablet_manager.cc:1151] Deleting tablet 00f34026fd2f4575bb86a9dadb733fe0
0317 21:19:07.624448 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.624728 (+ 280us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1784783}
I20260317 21:19:07.625522 7791 ts_tablet_manager.cc:1929] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260317 21:19:07.625514 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e already in progress: creating tablet
I20260317 21:19:07.625954 7869 tablet_replica.cc:333] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:07.626070 7791 log.cc:1199] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/35023195b9094d58827921a1dcb2ff05
W20260317 21:19:07.626183 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 5314c562bc9d457a8e1f4930e8d3a7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5314c562bc9d457a8e1f4930e8d3a7ff already in progress: creating tablet
I20260317 21:19:07.626776 7791 ts_tablet_manager.cc:1950] T 35023195b9094d58827921a1dcb2ff05 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
W20260317 21:19:07.626766 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet ae1b2a549af6485ca3faee3f0ad845a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae1b2a549af6485ca3faee3f0ad845a2 already in progress: creating tablet
W20260317 21:19:07.627869 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 00f34026fd2f4575bb86a9dadb733fe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00f34026fd2f4575bb86a9dadb733fe0 already in progress: creating tablet
I20260317 21:19:07.628686 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 226) took 2426 ms. Trace:
I20260317 21:19:07.629009 7791 rpcz_store.cc:276] 0317 21:19:05.202482 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.202663 (+ 181us) service_pool.cc:225] Handling call
0317 21:19:07.579281 (+2376618us) ts_tablet_manager.cc:1151] Deleting tablet 35023195b9094d58827921a1dcb2ff05
0317 21:19:07.579321 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.605096 (+ 25775us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.625293 (+ 20197us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.628594 (+ 3301us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2375790}
I20260317 21:19:07.630259 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 228) took 2163 ms. Trace:
I20260317 21:19:07.630573 7791 rpcz_store.cc:276] 0317 21:19:05.466882 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.467001 (+ 119us) service_pool.cc:225] Handling call
0317 21:19:07.629919 (+2162918us) ts_tablet_manager.cc:1151] Deleting tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e
0317 21:19:07.629961 (+ 42us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.630209 (+ 248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2155065}
I20260317 21:19:07.631649 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 232) took 1996 ms. Trace:
I20260317 21:19:07.631946 7791 rpcz_store.cc:276] 0317 21:19:05.635395 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.635638 (+ 243us) service_pool.cc:225] Handling call
0317 21:19:07.631350 (+1995712us) ts_tablet_manager.cc:1151] Deleting tablet 5314c562bc9d457a8e1f4930e8d3a7ff
0317 21:19:07.631387 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.631604 (+ 217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1994466}
I20260317 21:19:07.633221 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 35023195b9094d58827921a1dcb2ff05 (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]) successfully deleted
I20260317 21:19:07.633494 7038 raft_consensus.cc:2468] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
W20260317 21:19:07.633947 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f0545b61d2d434ea4e51a8f2c4c4d3e already in progress: creating tablet
I20260317 21:19:07.634526 7120 leader_election.cc:304] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:07.634972 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 5314c562bc9d457a8e1f4930e8d3a7ff because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5314c562bc9d457a8e1f4930e8d3a7ff already in progress: creating tablet
I20260317 21:19:07.635087 7467 tablet_bootstrap.cc:492] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:07.636438 7467 ts_tablet_manager.cc:1403] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.193s user 0.012s sys 0.004s
I20260317 21:19:07.638846 7467 raft_consensus.cc:359] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.639492 7467 raft_consensus.cc:385] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.639765 7467 raft_consensus.cc:740] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.640911 7791 tablet_replica.cc:333] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:07.641683 7791 raft_consensus.cc:2243] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:07.642367 7791 pending_rounds.cc:70] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:07.642596 7791 pending_rounds.cc:77] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412491931459584 op_type: NO_OP noop_request { }
I20260317 21:19:07.642946 7791 raft_consensus.cc:2889] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:07.643278 7791 raft_consensus.cc:2272] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.650486 8008 raft_consensus.cc:493] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.650992 8008 raft_consensus.cc:515] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.616882 7261 leader_election.cc:304] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:07.652902 8008 leader_election.cc:290] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.619633 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:07.654109 7326 raft_consensus.cc:2393] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:07.655555 8038 raft_consensus.cc:2749] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:07.656304 7791 ts_tablet_manager.cc:1916] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.657051 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:07.657150 7462 consensus_queue.cc:260] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.657810 7327 raft_consensus.cc:2393] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:07.660254 7462 ts_tablet_manager.cc:1434] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.108s user 0.006s sys 0.001s
I20260317 21:19:07.660676 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.662156 7120 leader_election.cc:304] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:07.662871 8008 raft_consensus.cc:3060] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.635504 7937 raft_consensus.cc:2804] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:07.666715 7937 raft_consensus.cc:697] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:07.667796 7937 consensus_queue.cc:237] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.668793 7229 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:07.671247 8008 raft_consensus.cc:2749] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:07.677922 7459 tablet_bootstrap.cc:492] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:07.678350 7459 ts_tablet_manager.cc:1403] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.007s
I20260317 21:19:07.681234 7459 raft_consensus.cc:359] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.683429 7459 raft_consensus.cc:385] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:07.683722 7459 raft_consensus.cc:740] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:07.685442 7791 ts_tablet_manager.cc:1929] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:07.685890 7791 log.cc:1199] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/2b287c83b77548f19546927b05905746
I20260317 21:19:07.686682 7791 ts_tablet_manager.cc:1950] T 2b287c83b77548f19546927b05905746 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.688763 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 234) took 1881 ms. Trace:
I20260317 21:19:07.688930 6675 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
I20260317 21:19:07.684294 7459 consensus_queue.cc:260] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.689148 7791 rpcz_store.cc:276] 0317 21:19:05.807348 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.807498 (+ 150us) service_pool.cc:225] Handling call
0317 21:19:07.640746 (+1833248us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:19:07.640787 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.676820 (+ 36033us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.685227 (+ 8407us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.688706 (+ 3479us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1824503}
I20260317 21:19:07.690398 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 235) took 1882 ms. Trace:
W20260317 21:19:07.690511 6675 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:45181 (request call id 62) took 760 ms (client timeout 1000 ms). Trace:
I20260317 21:19:07.690689 7791 rpcz_store.cc:276] 0317 21:19:05.808117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.809750 (+ 1633us) service_pool.cc:225] Handling call
0317 21:19:07.690088 (+1880338us) ts_tablet_manager.cc:1151] Deleting tablet ae1b2a549af6485ca3faee3f0ad845a2
0317 21:19:07.690125 (+ 37us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.690350 (+ 225us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1879645}
I20260317 21:19:07.690922 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 2b287c83b77548f19546927b05905746 (table default.Table_e4715bec3faa471cbdcb63c57766180a [id=ff96f5df47e0414290054b29ed53479a]) successfully deleted
W20260317 21:19:07.690804 6675 rpcz_store.cc:269] 0317 21:19:06.929942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.930075 (+ 133us) service_pool.cc:225] Handling call
0317 21:19:07.690450 (+760375us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260317 21:19:07.691687 7459 ts_tablet_manager.cc:1434] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.013s user 0.008s sys 0.000s
I20260317 21:19:07.691761 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 237) took 1798 ms. Trace:
W20260317 21:19:07.692034 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet ae1b2a549af6485ca3faee3f0ad845a2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ae1b2a549af6485ca3faee3f0ad845a2 already in progress: creating tablet
I20260317 21:19:07.692121 7791 rpcz_store.cc:276] 0317 21:19:05.915070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.915829 (+ 759us) service_pool.cc:225] Handling call
0317 21:19:07.691433 (+1775604us) ts_tablet_manager.cc:1151] Deleting tablet 00f34026fd2f4575bb86a9dadb733fe0
0317 21:19:07.691471 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.691713 (+ 242us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1760751}
I20260317 21:19:07.692610 7459 tablet_bootstrap.cc:492] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:07.693310 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 240) took 1423 ms. Trace:
W20260317 21:19:07.693375 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 00f34026fd2f4575bb86a9dadb733fe0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00f34026fd2f4575bb86a9dadb733fe0 already in progress: creating tablet
I20260317 21:19:07.693598 7791 rpcz_store.cc:276] 0317 21:19:06.270415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.270560 (+ 145us) service_pool.cc:225] Handling call
0317 21:19:07.692983 (+1422423us) ts_tablet_manager.cc:1151] Deleting tablet cef932d50d4544ad9c5ca245bc006f29
0317 21:19:07.693022 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.693260 (+ 238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1421818}
I20260317 21:19:07.694495 7791 tablet_replica.cc:333] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:07.694854 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet cef932d50d4544ad9c5ca245bc006f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef932d50d4544ad9c5ca245bc006f29 already in progress: creating tablet
I20260317 21:19:07.695163 7791 raft_consensus.cc:2243] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.695634 7791 raft_consensus.cc:2272] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.695806 6675 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_3c680f76c8ca4c0298b9f1e3487081f5" } new_table_name: "default.Table_5704b94c19d8479db890552467232ffb" modify_external_catalogs: true
I20260317 21:19:07.697325 7791 ts_tablet_manager.cc:1916] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.703831 8028 raft_consensus.cc:493] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.704315 8028 raft_consensus.cc:515] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.705389 6673 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 275e9cd7f14040d98eb85a6bb1010c2c
I20260317 21:19:07.706383 8028 leader_election.cc:290] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:07.707155 6673 catalog_manager.cc:6741] ReplaceTablet: tablet 275e9cd7f14040d98eb85a6bb1010c2c deleted and replaced by tablet 76bd0f5369cf400299039e4631ed11b0
I20260317 21:19:07.707499 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd0f5369cf400299039e4631ed11b0) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:07.709465 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "275e9cd7f14040d98eb85a6bb1010c2c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:07.710983 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd0f5369cf400299039e4631ed11b0) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:07.712111 6971 leader_election.cc:343] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.714339 7459 tablet_bootstrap.cc:654] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.715546 7791 ts_tablet_manager.cc:1929] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:07.715975 7791 log.cc:1199] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/59fce28afbeb4d8eadf63f9a97d1183d
I20260317 21:19:07.716605 7791 ts_tablet_manager.cc:1950] T 59fce28afbeb4d8eadf63f9a97d1183d P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.718405 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 241) took 1197 ms. Trace:
I20260317 21:19:07.718719 7791 rpcz_store.cc:276] 0317 21:19:06.520993 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.521153 (+ 160us) service_pool.cc:225] Handling call
0317 21:19:07.694366 (+1173213us) ts_tablet_manager.cc:1151] Deleting tablet 59fce28afbeb4d8eadf63f9a97d1183d
0317 21:19:07.694401 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.705213 (+ 10812us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.715181 (+ 9968us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.718348 (+ 3167us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1172661}
I20260317 21:19:07.720274 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 242) took 1192 ms. Trace:
I20260317 21:19:07.720351 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 59fce28afbeb4d8eadf63f9a97d1183d (table default.Table_6d4e07837b8945f981be0ae07f5b4ea0 [id=2e0f6bf62c664a6cb3f2bdad9a74e11b]) successfully deleted
I20260317 21:19:07.720578 7791 rpcz_store.cc:276] 0317 21:19:06.528013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.528166 (+ 153us) service_pool.cc:225] Handling call
0317 21:19:07.719821 (+1191655us) ts_tablet_manager.cc:1151] Deleting tablet b9d1076839894509805be59b751aa847
0317 21:19:07.719963 (+ 142us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.720225 (+ 262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1191101}
I20260317 21:19:07.721717 7791 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 243) took 1130 ms. Trace:
W20260317 21:19:07.721956 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet b9d1076839894509805be59b751aa847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9d1076839894509805be59b751aa847 already in progress: creating tablet
I20260317 21:19:07.722023 7791 rpcz_store.cc:276] 0317 21:19:06.591618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.591773 (+ 155us) service_pool.cc:225] Handling call
0317 21:19:07.721410 (+1129637us) ts_tablet_manager.cc:1151] Deleting tablet d4afbb1ab93a44d9ab5795b72444c690
0317 21:19:07.721450 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:07.721673 (+ 223us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1129064}
I20260317 21:19:07.722304 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_76d879b3858f45bab280f507b3ecec0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.722914 7791 tablet_replica.cc:333] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:07.723282 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet d4afbb1ab93a44d9ab5795b72444c690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4afbb1ab93a44d9ab5795b72444c690 already in progress: creating tablet
I20260317 21:19:07.723573 7791 raft_consensus.cc:2243] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.724030 7791 raft_consensus.cc:2272] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:07.724162 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_76d879b3858f45bab280f507b3ecec0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.726156 7791 ts_tablet_manager.cc:1916] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.726950 6668 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
I20260317 21:19:07.727691 6668 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_66c25b9b83724e718f148f260ec5f672 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 2 got 0
I20260317 21:19:07.729174 7467 consensus_queue.cc:260] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.730412 6670 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]). Expected version 3 got 0
W20260317 21:19:07.735899 6670 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 55) took 1334 ms (client timeout 1000 ms). Trace:
I20260317 21:19:07.736373 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 4af622a9190148ee80b2d5d3d7a92a20
W20260317 21:19:07.736929 6670 rpcz_store.cc:269] 0317 21:19:06.401173 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.425680 (+ 24507us) service_pool.cc:225] Handling call
0317 21:19:06.557387 (+131707us) spinlock_profiling.cc:243] Waited 130 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:07.735821 (+1178434us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":2029,"reported_tablets":3,"spinlock_wait_cycles":298026240}
I20260317 21:19:07.738153 6671 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 317) took 1207 ms. Trace:
I20260317 21:19:07.743011 6669 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
I20260317 21:19:07.744951 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 4af622a9190148ee80b2d5d3d7a92a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:07.750814 6675 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 21e607674c734eaea207ed454097791d consensus state has no leader
I20260317 21:19:07.738495 6671 rpcz_store.cc:276] 0317 21:19:06.530326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.530425 (+ 99us) service_pool.cc:225] Handling call
0317 21:19:06.558481 (+ 28056us) spinlock_profiling.cc:243] Waited 27.3 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1386 00007fc05bed7024 00007fc05bed6fb2 00007fc05bed6ee1 00007fc05bed6e5c 00007fc05bed6112 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:06.560147 (+ 1666us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:06.560632 (+ 485us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:06.563610 (+ 2978us) catalog_manager.cc:2824] Locking tablets
0317 21:19:07.682973 (+1119363us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:07.735991 (+ 53018us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:19:07.736189 (+ 198us) catalog_manager.cc:2872] Committing in-memory state
0317 21:19:07.736286 (+ 97us) catalog_manager.cc:2876] Aborting table tasks
0317 21:19:07.738097 (+ 1811us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:07.684865 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:07.685286 (+ 421us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:07.685308 (+ 22us) write_op.cc:435] Acquired schema lock
0317 21:19:07.685320 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:07.685635 (+ 315us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:07.685669 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:07.685694 (+ 25us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:07.685988 (+ 294us) tablet.cc:700] Row locks acquired
0317 21:19:07.686002 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:07.686121 (+ 119us) write_op.cc:270] Start()
0317 21:19:07.686204 (+ 83us) write_op.cc:276] Timestamp: P: 1773782347686074 usec, L: 0
0317 21:19:07.686222 (+ 18us) op_driver.cc:348] REPLICATION: starting
0317 21:19:07.686736 (+ 514us) log.cc:844] Serialized 728 byte log entry
0317 21:19:07.731670 (+ 44934us) op_driver.cc:464] REPLICATION: finished
0317 21:19:07.731948 (+ 278us) write_op.cc:301] APPLY: starting
0317 21:19:07.732004 (+ 56us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:07.732114 (+ 110us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:07.732209 (+ 95us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:07.732482 (+ 273us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:07.732629 (+ 147us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:07.733379 (+ 750us) write_op.cc:312] APPLY: finished
0317 21:19:07.733596 (+ 217us) log.cc:844] Serialized 29 byte log entry
0317 21:19:07.733767 (+ 171us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:07.733853 (+ 86us) write_op.cc:454] Released schema lock
0317 21:19:07.733886 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:07.734325 (+ 439us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:07.734353 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":62843904,"child_traces":[["op",{"apply.queue_time_us":131,"apply.run_cpu_time_us":1677,"apply.run_wall_time_us":2304,"num_ops":2,"prepare.queue_time_us":143,"prepare.run_cpu_time_us":2103,"prepare.run_wall_time_us":3238,"replication_time_us":45345}]]}
I20260317 21:19:07.754418 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 4af622a9190148ee80b2d5d3d7a92a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:07.691417 8038 raft_consensus.cc:493] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.756893 8028 consensus_queue.cc:1048] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:07.757344 8038 raft_consensus.cc:515] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.708118 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd0f5369cf400299039e4631ed11b0) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:07.708446 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "275e9cd7f14040d98eb85a6bb1010c2c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:07.759416 8038 leader_election.cc:290] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:07.759500 7327 raft_consensus.cc:2468] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:07.759774 7791 ts_tablet_manager.cc:1929] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:07.760129 7791 log.cc:1199] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/275e9cd7f14040d98eb85a6bb1010c2c
I20260317 21:19:07.739241 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 4af622a9190148ee80b2d5d3d7a92a20 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:07.760821 7791 ts_tablet_manager.cc:1950] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.760691 8008 raft_consensus.cc:493] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.761008 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b066a6a5c864beb8da59bd97bacd309" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:07.761152 8008 raft_consensus.cc:515] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.761653 6973 leader_election.cc:304] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:07.762219 6973 raft_consensus.cc:2686] T 275e9cd7f14040d98eb85a6bb1010c2c P d3cb3b49e3e94537a6e434ea4e037c3f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:19:07.763803 8008 leader_election.cc:290] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:07.764976 7261 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:07.765805 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 275e9cd7f14040d98eb85a6bb1010c2c (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]) successfully deleted
I20260317 21:19:07.766425 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
W20260317 21:19:07.766415 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 4af622a9190148ee80b2d5d3d7a92a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af622a9190148ee80b2d5d3d7a92a20 already in progress: creating tablet
I20260317 21:19:07.766459 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:07.661178 7462 tablet_bootstrap.cc:492] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:07.767150 7327 raft_consensus.cc:2468] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:07.767632 6672 catalog_manager.cc:5671] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:07.768395 7119 leader_election.cc:304] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:07.769188 8008 raft_consensus.cc:2804] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:07.769167 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_66c25b9b83724e718f148f260ec5f672 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 2 got 0
I20260317 21:19:07.769582 8008 raft_consensus.cc:493] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:07.769901 8008 raft_consensus.cc:3060] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.770179 6672 catalog_manager.cc:5671] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:07.771545 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:07.772217 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:07.773442 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.774464 7038 raft_consensus.cc:2468] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:07.773254 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:07.775451 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_9a3ec369ff814877a0ff250609d2803e [id=81dc7f934d2346cd8ad83e5d718c5d7d]). Expected version 1 got 0
W20260317 21:19:07.775725 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 21e607674c734eaea207ed454097791d (table default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 21e607674c734eaea207ed454097791d consensus state has no leader
I20260317 21:19:07.779920 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 32be4818a67a41c998f04d7cedf7356e (DEFAULT_TABLE table=default.Table_f9ab9898a12b4ffeb9a2c721ee8f05c3 [id=f1fd5b396a47401da6d5923002e072c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.779932 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b066a6a5c864beb8da59bd97bacd309" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:07.781013 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32be4818a67a41c998f04d7cedf7356e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.780797 7038 raft_consensus.cc:2468] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:07.782400 7264 leader_election.cc:304] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:07.783933 8038 raft_consensus.cc:2804] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:07.784375 8038 raft_consensus.cc:493] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:07.784861 8038 raft_consensus.cc:3060] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.788882 7869 raft_consensus.cc:2243] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
W20260317 21:19:07.791106 6667 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6caf05c79a5f470e8b7796d0e85e36f1 consensus state has no leader
I20260317 21:19:07.794564 7869 raft_consensus.cc:2272] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:07.794719 6668 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 74) took 1247 ms (client timeout 1000 ms). Trace:
W20260317 21:19:07.795154 6668 rpcz_store.cc:269] 0317 21:19:06.547286 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.547387 (+ 101us) service_pool.cc:225] Handling call
0317 21:19:06.551364 (+ 3977us) spinlock_profiling.cc:243] Waited 1.85 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:07.794644 (+1243280us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":5,"spinlock_wait_cycles":4261120}
I20260317 21:19:07.797441 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 0eee8b5588c24661889c593797c94269 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:07.798180 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:07.799502 7791 tablet_replica.cc:333] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
W20260317 21:19:07.799731 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 0eee8b5588c24661889c593797c94269 because the tablet was not found. No further retry: Not found: Tablet not found: 0eee8b5588c24661889c593797c94269
I20260317 21:19:07.800310 7791 raft_consensus.cc:2243] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.800889 8106 tablet_replica.cc:333] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:07.800844 7791 raft_consensus.cc:2272] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.802718 7791 ts_tablet_manager.cc:1916] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.805691 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 32be4818a67a41c998f04d7cedf7356e (DEFAULT_TABLE table=default.Table_f9ab9898a12b4ffeb9a2c721ee8f05c3 [id=f1fd5b396a47401da6d5923002e072c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.806699 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32be4818a67a41c998f04d7cedf7356e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.806752 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 32be4818a67a41c998f04d7cedf7356e (DEFAULT_TABLE table=default.Table_f9ab9898a12b4ffeb9a2c721ee8f05c3 [id=f1fd5b396a47401da6d5923002e072c6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.807742 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32be4818a67a41c998f04d7cedf7356e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.801769 8106 raft_consensus.cc:2243] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.812546 8106 raft_consensus.cc:2272] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.817404 8106 ts_tablet_manager.cc:1916] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.821023 7015 tablet_service.cc:1511] Processing CreateTablet for tablet e2c52f4843234e8584a66e5e8e065667 (DEFAULT_TABLE table=default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.822155 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c52f4843234e8584a66e5e8e065667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.823035 7305 tablet_service.cc:1511] Processing CreateTablet for tablet e2c52f4843234e8584a66e5e8e065667 (DEFAULT_TABLE table=default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.823273 6670 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_48c6d19344564ada866cf36024f05a91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:07.824095 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c52f4843234e8584a66e5e8e065667. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:07.825102 6670 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48c6d19344564ada866cf36024f05a91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.825716 7010 tablet_service.cc:1511] Processing CreateTablet for tablet d69e14125d7d4412869c20759acc2336 (DEFAULT_TABLE table=default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.827136 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69e14125d7d4412869c20759acc2336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.816344 7155 tablet_service.cc:1511] Processing CreateTablet for tablet e2c52f4843234e8584a66e5e8e065667 (DEFAULT_TABLE table=default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.831969 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2c52f4843234e8584a66e5e8e065667. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.834924 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_03cf469e6cf349c4a7dea286f9ae78a8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:07.836539 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_03cf469e6cf349c4a7dea286f9ae78a8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.836689 8028 consensus_queue.cc:1048] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:07.860136 7298 tablet_service.cc:1511] Processing CreateTablet for tablet d69e14125d7d4412869c20759acc2336 (DEFAULT_TABLE table=default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.861423 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69e14125d7d4412869c20759acc2336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.817313 7467 ts_tablet_manager.cc:1434] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.181s user 0.006s sys 0.000s
I20260317 21:19:07.830176 7152 tablet_service.cc:1511] Processing CreateTablet for tablet d69e14125d7d4412869c20759acc2336 (DEFAULT_TABLE table=default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.863831 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d69e14125d7d4412869c20759acc2336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.868158 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d86c06b3eb2849ffb26839db4313557e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:07.869812 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d86c06b3eb2849ffb26839db4313557e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:07.873422 7467 tablet_bootstrap.cc:492] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:07.874187 7869 ts_tablet_manager.cc:1916] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.876046 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 646a115e133b43eebc1674a862ab7838 (DEFAULT_TABLE table=default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.876472 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 646a115e133b43eebc1674a862ab7838 (DEFAULT_TABLE table=default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.877569 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646a115e133b43eebc1674a862ab7838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.877656 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646a115e133b43eebc1674a862ab7838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.879168 8008 raft_consensus.cc:515] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.881824 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:07.882493 7038 raft_consensus.cc:3060] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.887567 8038 raft_consensus.cc:515] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.890627 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b066a6a5c864beb8da59bd97bacd309" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:07.891215 7036 raft_consensus.cc:3060] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:07.891826 7791 ts_tablet_manager.cc:1929] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:07.892176 7791 log.cc:1199] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/78e2e20d320547b3bb1e03923de52822
I20260317 21:19:07.892956 7791 ts_tablet_manager.cc:1950] T 78e2e20d320547b3bb1e03923de52822 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.901083 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 78e2e20d320547b3bb1e03923de52822 (table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]) successfully deleted
I20260317 21:19:07.906747 7036 raft_consensus.cc:2468] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:07.907876 7264 leader_election.cc:304] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:07.908629 8097 raft_consensus.cc:2804] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:07.922348 6674 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:07.923820 6674 catalog_manager.cc:6741] ReplaceTablet: tablet 5b066a6a5c864beb8da59bd97bacd309 deleted and replaced by tablet 4f33574134934f048632aae269244368
I20260317 21:19:07.924192 6674 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39038 (request call id 318) took 1295 ms. Trace:
I20260317 21:19:07.924880 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f33574134934f048632aae269244368) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:07.927191 7462 tablet_bootstrap.cc:654] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:07.928355 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f33574134934f048632aae269244368) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:07.931327 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f33574134934f048632aae269244368) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:07.924465 6674 rpcz_store.cc:276] 0317 21:19:06.629239 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.629428 (+ 189us) service_pool.cc:225] Handling call
0317 21:19:07.924134 (+1294706us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:07.725726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:07.726148 (+ 422us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:07.726176 (+ 28us) write_op.cc:435] Acquired schema lock
0317 21:19:07.726190 (+ 14us) tablet.cc:661] Decoding operations
0317 21:19:07.726628 (+ 438us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:07.726665 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:07.726686 (+ 21us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:07.726884 (+ 198us) tablet.cc:700] Row locks acquired
0317 21:19:07.726898 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:07.727017 (+ 119us) write_op.cc:270] Start()
0317 21:19:07.727099 (+ 82us) write_op.cc:276] Timestamp: P: 1773782347726965 usec, L: 0
0317 21:19:07.727115 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:07.727628 (+ 513us) log.cc:844] Serialized 625 byte log entry
0317 21:19:07.901226 (+173598us) op_driver.cc:464] REPLICATION: finished
0317 21:19:07.915930 (+ 14704us) write_op.cc:301] APPLY: starting
0317 21:19:07.915993 (+ 63us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:07.916094 (+ 101us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:07.916206 (+ 112us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:07.916577 (+ 371us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:07.916796 (+ 219us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:07.916839 (+ 43us) write_op.cc:312] APPLY: finished
0317 21:19:07.917041 (+ 202us) log.cc:844] Serialized 29 byte log entry
0317 21:19:07.920368 (+ 3327us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:07.920442 (+ 74us) write_op.cc:454] Released schema lock
0317 21:19:07.920483 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":14489,"num_ops":2,"prepare.queue_time_us":12680,"prepare.run_cpu_time_us":2014,"prepare.run_wall_time_us":3154,"replication_time_us":174018}]]}
I20260317 21:19:07.934854 7791 tablet_replica.cc:333] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:07.935796 7791 raft_consensus.cc:2243] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:07.936369 7791 raft_consensus.cc:2272] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:07.938735 7791 ts_tablet_manager.cc:1916] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:07.942102 7038 raft_consensus.cc:2468] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:07.945448 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 646a115e133b43eebc1674a862ab7838 (DEFAULT_TABLE table=default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.946938 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 646a115e133b43eebc1674a862ab7838. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.950417 7120 leader_election.cc:304] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:07.951869 8012 raft_consensus.cc:2804] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:07.952410 8012 raft_consensus.cc:697] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:07.953300 8012 consensus_queue.cc:237] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.957283 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f2bf8c9535e94059a16a46b599e39416" } modify_external_catalogs: true
I20260317 21:19:07.957811 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f2bf8c9535e94059a16a46b599e39416" } modify_external_catalogs: true
I20260317 21:19:07.958339 7791 ts_tablet_manager.cc:1929] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:07.958701 7791 log.cc:1199] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:07.959410 7791 ts_tablet_manager.cc:1950] T 5b066a6a5c864beb8da59bd97bacd309 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:07.962157 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 5b066a6a5c864beb8da59bd97bacd309 (table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]) successfully deleted
I20260317 21:19:07.963255 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_67179be887be4140976c9d859cba3a88" } new_table_name: "default.Table_525a8435bbe14a13a70b022bb2c07f59" modify_external_catalogs: true
I20260317 21:19:07.980592 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 17610bdee98f49c4b1d8b5481867c6e8 (DEFAULT_TABLE table=default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.988703 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 17610bdee98f49c4b1d8b5481867c6e8 (DEFAULT_TABLE table=default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:07.989949 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17610bdee98f49c4b1d8b5481867c6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:07.996965 8028 raft_consensus.cc:493] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:07.997558 8028 raft_consensus.cc:515] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:07.999476 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 17610bdee98f49c4b1d8b5481867c6e8 (DEFAULT_TABLE table=default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.000573 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17610bdee98f49c4b1d8b5481867c6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.007637 8028 leader_election.cc:290] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.008265 8028 raft_consensus.cc:493] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.008818 8028 raft_consensus.cc:515] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.012172 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a978fa28a31b4d589031c77526b62b8e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:08.013659 6971 leader_election.cc:343] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.016906 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
W20260317 21:19:08.028342 8079 lock_manager.cc:585] Waited 1 seconds to obtain row lock on key '<redacted>' (key hash 9073464951982539180) tablet 00000000000000000000000000000000 cur holder 0x7b5800050700
I20260317 21:19:08.030359 8038 leader_election.cc:290] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:08.032367 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 2152e5cbd2ce4159b0901ba2ffe41f40 (DEFAULT_TABLE table=default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.036485 7008 tablet_service.cc:1511] Processing CreateTablet for tablet aa814dcc2499482eaf0afc5d95bb40f1 (DEFAULT_TABLE table=default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.035808 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b066a6a5c864beb8da59bd97bacd309" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:08.050603 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17610bdee98f49c4b1d8b5481867c6e8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.061228 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2152e5cbd2ce4159b0901ba2ffe41f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.061793 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa814dcc2499482eaf0afc5d95bb40f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.036464 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 2152e5cbd2ce4159b0901ba2ffe41f40 (DEFAULT_TABLE table=default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:08.039415 7261 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.069528 8114 raft_consensus.cc:493] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.070091 8114 raft_consensus.cc:515] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.072182 8114 leader_election.cc:290] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:08.074313 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8567ae10464694a1128ba2eb1d8ee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:08.075924 7261 leader_election.cc:343] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.052863 7462 tablet_bootstrap.cc:492] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:08.061506 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:08.077108 7462 ts_tablet_manager.cc:1403] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.416s user 0.009s sys 0.003s
I20260317 21:19:08.077207 7328 raft_consensus.cc:2468] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:08.079515 7462 raft_consensus.cc:359] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.080238 7462 raft_consensus.cc:385] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.080493 7462 raft_consensus.cc:740] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.081329 7459 tablet_bootstrap.cc:492] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:08.081755 7459 ts_tablet_manager.cc:1403] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.389s user 0.083s sys 0.103s
I20260317 21:19:08.082916 8028 leader_election.cc:290] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.085284 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8567ae10464694a1128ba2eb1d8ee1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:08.091910 7459 raft_consensus.cc:359] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.100991 7459 raft_consensus.cc:385] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.101728 7459 raft_consensus.cc:740] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.102439 7459 consensus_queue.cc:260] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.105870 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 9c3b084b04e944d7b919ec1965742e9f (DEFAULT_TABLE table=default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.106387 7459 ts_tablet_manager.cc:1434] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.024s user 0.007s sys 0.001s
I20260317 21:19:08.106854 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3b084b04e944d7b919ec1965742e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.107360 7459 tablet_bootstrap.cc:492] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:08.113735 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 83832.1us
I20260317 21:19:08.086055 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2152e5cbd2ce4159b0901ba2ffe41f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.118422 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 2152e5cbd2ce4159b0901ba2ffe41f40 (DEFAULT_TABLE table=default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.119210 7869 ts_tablet_manager.cc:1929] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:08.119674 7869 log.cc:1199] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/35023195b9094d58827921a1dcb2ff05
I20260317 21:19:08.119699 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2152e5cbd2ce4159b0901ba2ffe41f40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.086381 6973 leader_election.cc:304] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:08.121558 7264 leader_election.cc:343] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.121971 7264 leader_election.cc:304] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:08.122625 8038 raft_consensus.cc:2749] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:08.123067 8028 raft_consensus.cc:2804] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:08.123479 8028 raft_consensus.cc:493] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:08.123889 8028 raft_consensus.cc:3060] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:08.126760 7869 ts_tablet_manager.cc:1950] T 35023195b9094d58827921a1dcb2ff05 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:08.132403 7459 tablet_bootstrap.cc:654] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:08.087869 8097 raft_consensus.cc:697] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:08.133353 8097 consensus_queue.cc:237] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.141215 7462 consensus_queue.cc:260] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.143514 7462 ts_tablet_manager.cc:1434] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.066s user 0.005s sys 0.000s
I20260317 21:19:08.144868 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 8744f6237e154dd99bc4c7e54822f7be
I20260317 21:19:08.145994 7462 tablet_bootstrap.cc:492] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:08.146306 6666 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 325) took 1349 ms. Trace:
I20260317 21:19:08.151966 6673 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_03e967dfb6ec4350932c7c69cec172b3" } new_table_name: "default.Table_b12dceef6882429eab39af906714c5e1" modify_external_catalogs: true
I20260317 21:19:08.146598 6666 rpcz_store.cc:276] 0317 21:19:06.796627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.796771 (+ 144us) service_pool.cc:225] Handling call
0317 21:19:06.812964 (+ 16193us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1386 00007fc05bed7024 00007fc05bed6fb2 00007fc05bed6ee1 00007fc05bed6e5c 00007fc05bed6112 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:06.823553 (+ 10589us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:06.824262 (+ 709us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:06.824439 (+ 177us) catalog_manager.cc:2824] Locking tablets
0317 21:19:07.747663 (+923224us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:08.144458 (+396795us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:19:08.144694 (+ 236us) catalog_manager.cc:2872] Committing in-memory state
0317 21:19:08.144778 (+ 84us) catalog_manager.cc:2876] Aborting table tasks
0317 21:19:08.146250 (+ 1472us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:07.833753 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:07.845017 (+ 11264us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:07.845039 (+ 22us) write_op.cc:435] Acquired schema lock
0317 21:19:07.845052 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:07.845337 (+ 285us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:07.845371 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:07.845392 (+ 21us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:07.845664 (+ 272us) tablet.cc:700] Row locks acquired
0317 21:19:07.845677 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:07.845800 (+ 123us) write_op.cc:270] Start()
0317 21:19:07.845893 (+ 93us) write_op.cc:276] Timestamp: P: 1773782347845735 usec, L: 0
0317 21:19:07.845906 (+ 13us) op_driver.cc:348] REPLICATION: starting
0317 21:19:07.846404 (+ 498us) log.cc:844] Serialized 728 byte log entry
0317 21:19:08.139837 (+293433us) op_driver.cc:464] REPLICATION: finished
0317 21:19:08.141942 (+ 2105us) write_op.cc:301] APPLY: starting
0317 21:19:08.141989 (+ 47us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:08.142086 (+ 97us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:08.142199 (+ 113us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:08.142427 (+ 228us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:08.142582 (+ 155us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:08.142637 (+ 55us) write_op.cc:312] APPLY: finished
0317 21:19:08.142804 (+ 167us) log.cc:844] Serialized 29 byte log entry
0317 21:19:08.142943 (+ 139us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:08.143007 (+ 64us) write_op.cc:454] Released schema lock
0317 21:19:08.143037 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":34684288,"child_traces":[["op",{"apply.queue_time_us":1980,"apply.run_cpu_time_us":1334,"apply.run_wall_time_us":7362,"num_ops":2,"prepare.queue_time_us":83966,"prepare.run_cpu_time_us":2151,"prepare.run_wall_time_us":12968,"replication_time_us":293860}]]}
I20260317 21:19:08.161154 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 8744f6237e154dd99bc4c7e54822f7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:08.162668 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 8744f6237e154dd99bc4c7e54822f7be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8744f6237e154dd99bc4c7e54822f7be already in progress: creating tablet
I20260317 21:19:08.162994 6668 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
I20260317 21:19:08.163668 6668 catalog_manager.cc:5671] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
W20260317 21:19:08.166357 6973 leader_election.cc:343] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:08.166808 6973 leader_election.cc:304] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:08.172886 7459 tablet_bootstrap.cc:492] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:08.173010 6668 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:08.173336 7459 ts_tablet_manager.cc:1403] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.066s user 0.010s sys 0.000s
I20260317 21:19:08.174366 6668 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 3 got 0
I20260317 21:19:08.175001 8060 raft_consensus.cc:2749] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:08.131175 8028 raft_consensus.cc:515] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.176074 7459 raft_consensus.cc:359] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.179049 6666 master_service.cc:954] ReplaceTablet: received request to replace tablet 70560dde990843e0b7c74ae3206c58e4 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:08.179282 6669 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8] alter complete (version 3)
I20260317 21:19:08.178107 7459 raft_consensus.cc:385] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.181943 7459 raft_consensus.cc:740] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.183014 7459 consensus_queue.cc:260] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.182359 8028 leader_election.cc:290] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:08.185513 6669 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 56) took 844 ms (client timeout 1000 ms). Trace:
I20260317 21:19:08.185792 7459 ts_tablet_manager.cc:1434] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.012s user 0.006s sys 0.000s
I20260317 21:19:08.186662 7459 tablet_bootstrap.cc:492] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:08.192382 7459 tablet_bootstrap.cc:654] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:08.196751 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet 78e2e20d320547b3bb1e03923de52822 with delete_type TABLET_DATA_DELETED (replaced by tablet 203f33157a0d4b82aab44da452aa46dd) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.197793 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet a978fa28a31b4d589031c77526b62b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:08.198168 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 78e2e20d320547b3bb1e03923de52822 because the tablet was not found. No further retry: Not found: Tablet not found: 78e2e20d320547b3bb1e03923de52822
I20260317 21:19:08.198993 7791 tablet_replica.cc:333] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:08.199700 7791 raft_consensus.cc:2243] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:08.200194 7791 raft_consensus.cc:2272] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:08.201902 7791 ts_tablet_manager.cc:1916] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:08.185932 6669 rpcz_store.cc:269] 0317 21:19:07.341035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:07.341833 (+ 798us) service_pool.cc:225] Handling call
0317 21:19:08.185452 (+843619us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:07.846928 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:07.847313 (+ 385us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:07.847335 (+ 22us) write_op.cc:435] Acquired schema lock
0317 21:19:07.847349 (+ 14us) tablet.cc:661] Decoding operations
0317 21:19:07.847716 (+ 367us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:07.847839 (+ 123us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:07.847865 (+ 26us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:07.848016 (+ 151us) tablet.cc:700] Row locks acquired
0317 21:19:07.848027 (+ 11us) write_op.cc:260] PREPARE: finished
0317 21:19:07.848135 (+ 108us) write_op.cc:270] Start()
0317 21:19:07.848215 (+ 80us) write_op.cc:276] Timestamp: P: 1773782347848077 usec, L: 0
0317 21:19:07.848239 (+ 24us) op_driver.cc:348] REPLICATION: starting
0317 21:19:07.852877 (+ 4638us) log.cc:844] Serialized 356 byte log entry
0317 21:19:08.150609 (+297732us) op_driver.cc:464] REPLICATION: finished
0317 21:19:08.156173 (+ 5564us) write_op.cc:301] APPLY: starting
0317 21:19:08.156237 (+ 64us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:08.156327 (+ 90us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:08.156434 (+ 107us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:08.156609 (+ 175us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:08.156781 (+ 172us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:08.156826 (+ 45us) write_op.cc:312] APPLY: finished
0317 21:19:08.157007 (+ 181us) log.cc:844] Serialized 23 byte log entry
0317 21:19:08.157168 (+ 161us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:08.157232 (+ 64us) write_op.cc:454] Released schema lock
0317 21:19:08.157265 (+ 33us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:08.157524 (+ 259us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:08.157550 (+ 26us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":7,"child_traces":[["op",{"apply.queue_time_us":5402,"apply.run_cpu_time_us":1326,"apply.run_wall_time_us":1326,"num_ops":1,"prepare.queue_time_us":87108,"prepare.run_cpu_time_us":2139,"prepare.run_wall_time_us":6179,"replication_time_us":302272}]]}
I20260317 21:19:08.205380 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.206660 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 2b287c83b77548f19546927b05905746 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:55 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.218230 7459 tablet_bootstrap.cc:492] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:08.218672 7459 ts_tablet_manager.cc:1403] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.009s
I20260317 21:19:08.219888 8060 consensus_queue.cc:1048] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:08.221233 7459 raft_consensus.cc:359] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.221959 7459 raft_consensus.cc:385] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.222222 7459 raft_consensus.cc:740] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.222896 7459 consensus_queue.cc:260] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.225818 7459 ts_tablet_manager.cc:1434] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:08.229375 7159 tablet_service.cc:1511] Processing CreateTablet for tablet aa814dcc2499482eaf0afc5d95bb40f1 (DEFAULT_TABLE table=default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.230468 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa814dcc2499482eaf0afc5d95bb40f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.231734 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 9c3b084b04e944d7b919ec1965742e9f (DEFAULT_TABLE table=default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.233210 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 8744f6237e154dd99bc4c7e54822f7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.234509 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3b084b04e944d7b919ec1965742e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.237066 7459 tablet_bootstrap.cc:492] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:08.237480 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 5e080308f93448138a4846ab6bec7dfa (DEFAULT_TABLE table=default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.238427 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e080308f93448138a4846ab6bec7dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.245193 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 243) took 2019 ms. Trace:
I20260317 21:19:08.245581 7869 rpcz_store.cc:276] 0317 21:19:06.226262 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.238826 (+ 12564us) service_pool.cc:225] Handling call
0317 21:19:07.625815 (+1386989us) ts_tablet_manager.cc:1151] Deleting tablet 35023195b9094d58827921a1dcb2ff05
0317 21:19:07.625850 (+ 35us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.100040 (+474190us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:08.119001 (+ 18961us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:08.245130 (+126129us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1386321}
I20260317 21:19:08.247260 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 244) took 1978 ms. Trace:
I20260317 21:19:08.247586 7869 rpcz_store.cc:276] 0317 21:19:06.269339 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.273642 (+ 4303us) service_pool.cc:225] Handling call
0317 21:19:08.246538 (+1972896us) ts_tablet_manager.cc:1151] Deleting tablet cef932d50d4544ad9c5ca245bc006f29
0317 21:19:08.246869 (+ 331us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.247188 (+ 319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1943296}
W20260317 21:19:08.249127 8115 log.cc:927] Time spent T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Append to log took a long time: real 0.272s user 0.000s sys 0.000s
I20260317 21:19:08.248567 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 245) took 1718 ms. Trace:
I20260317 21:19:08.252936 7869 rpcz_store.cc:276] 0317 21:19:06.530160 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.531438 (+ 1278us) service_pool.cc:225] Handling call
0317 21:19:08.248288 (+1716850us) ts_tablet_manager.cc:1151] Deleting tablet b9d1076839894509805be59b751aa847
0317 21:19:08.248322 (+ 34us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.248523 (+ 201us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1716339}
I20260317 21:19:08.263238 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 35023195b9094d58827921a1dcb2ff05 (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]) successfully deleted
W20260317 21:19:08.264529 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet cef932d50d4544ad9c5ca245bc006f29 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cef932d50d4544ad9c5ca245bc006f29 already in progress: creating tablet
W20260317 21:19:08.265218 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet b9d1076839894509805be59b751aa847 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b9d1076839894509805be59b751aa847 already in progress: creating tablet
I20260317 21:19:08.120162 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:08.266906 8008 leader_election.cc:290] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.254701 7869 tablet_replica.cc:333] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
W20260317 21:19:08.268800 6971 leader_election.cc:343] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.269413 7869 raft_consensus.cc:2243] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:08.272117 7869 raft_consensus.cc:2272] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:08.273749 7119 consensus_peers.cc:597] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78e2e20d320547b3bb1e03923de52822. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:08.274288 7467 tablet_bootstrap.cc:654] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:08.274698 6971 consensus_peers.cc:597] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:08.275763 7869 ts_tablet_manager.cc:1916] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:08.277021 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 5e080308f93448138a4846ab6bec7dfa (DEFAULT_TABLE table=default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.288800 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e080308f93448138a4846ab6bec7dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.293133 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet f3b5825c77ac4d69a39978932051db8c
W20260317 21:19:08.303372 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4495961a559549629dedf8ce032abc9b consensus state has no leader
I20260317 21:19:08.307185 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:08.309063 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet f3b5825c77ac4d69a39978932051db8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:07 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.316349 7791 ts_tablet_manager.cc:1929] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:08.316879 7791 log.cc:1199] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:08.317626 7791 ts_tablet_manager.cc:1950] T a978fa28a31b4d589031c77526b62b8e P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:08.314683 6672 master_service.cc:954] ReplaceTablet: received request to replace tablet 440cb8b112044a53804357056f1226f1 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:08.334986 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 70560dde990843e0b7c74ae3206c58e4
I20260317 21:19:08.336318 6666 catalog_manager.cc:6741] ReplaceTablet: tablet 70560dde990843e0b7c74ae3206c58e4 deleted and replaced by tablet b03b54b989d04e0990ee3ccfe7a104a4
W20260317 21:19:08.336874 6673 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dde40a1c77854adcabc2fff3baaf4c81 consensus state has no leader
I20260317 21:19:08.338665 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 70560dde990843e0b7c74ae3206c58e4 with delete_type TABLET_DATA_DELETED (replaced by tablet b03b54b989d04e0990ee3ccfe7a104a4) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.342515 6669 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]). Expected version 2 got 0
I20260317 21:19:08.343466 6669 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 3 got 0
I20260317 21:19:08.344118 6669 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:08.344830 7869 ts_tablet_manager.cc:1929] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:08.345191 7869 log.cc:1199] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/2b287c83b77548f19546927b05905746
I20260317 21:19:08.345841 7869 ts_tablet_manager.cc:1950] T 2b287c83b77548f19546927b05905746 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:08.350495 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 35023195b9094d58827921a1dcb2ff05 with delete_type TABLET_DATA_DELETED (replaced by tablet bc2cd63423f844aea29ed3fd839f083e) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.351235 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_42a3f178b5b24f6388658b40093d97aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:08.110136 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 9c3b084b04e944d7b919ec1965742e9f (DEFAULT_TABLE table=default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.147249 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet 8744f6237e154dd99bc4c7e54822f7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.181392 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:08.354113 7328 raft_consensus.cc:3060] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:08.354480 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9c3b084b04e944d7b919ec1965742e9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.190140 7301 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d40549347f8463797d6791136883371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.190788 7304 tablet_service.cc:1558] Processing DeleteTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.117395 7298 tablet_service.cc:1511] Processing CreateTablet for tablet aa814dcc2499482eaf0afc5d95bb40f1 (DEFAULT_TABLE table=default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.357776 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aa814dcc2499482eaf0afc5d95bb40f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.223021 8106 ts_tablet_manager.cc:1929] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:08.360872 8106 log.cc:1199] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/275e9cd7f14040d98eb85a6bb1010c2c
I20260317 21:19:08.361657 8106 ts_tablet_manager.cc:1950] T 275e9cd7f14040d98eb85a6bb1010c2c P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:08.362170 7328 raft_consensus.cc:2468] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:08.363437 6973 leader_election.cc:304] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:08.363740 8106 tablet_replica.cc:333] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:08.237238 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 5e080308f93448138a4846ab6bec7dfa (DEFAULT_TABLE table=default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.248982 8038 raft_consensus.cc:493] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.274868 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:08.364928 7327 raft_consensus.cc:3060] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:08.364859 8038 raft_consensus.cc:515] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.367293 8038 leader_election.cc:290] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:08.302170 7299 tablet_service.cc:1558] Processing DeleteTablet for tablet f3b5825c77ac4d69a39978932051db8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:07 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.340276 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet 70560dde990843e0b7c74ae3206c58e4 with delete_type TABLET_DATA_DELETED (replaced by tablet b03b54b989d04e0990ee3ccfe7a104a4) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.371673 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e080308f93448138a4846ab6bec7dfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.374032 7327 raft_consensus.cc:2468] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:08.382174 8106 raft_consensus.cc:2243] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:08.383039 8106 pending_rounds.cc:70] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Trying to abort 1 pending ops.
I20260317 21:19:08.383302 8106 pending_rounds.cc:77] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412497981431808 op_type: NO_OP noop_request { }
I20260317 21:19:08.383751 8106 raft_consensus.cc:2889] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:08.384119 8106 raft_consensus.cc:2272] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:08.385160 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_42a3f178b5b24f6388658b40093d97aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:08.392912 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 275e9cd7f14040d98eb85a6bb1010c2c (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]) successfully deleted
W20260317 21:19:08.397136 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 4af622a9190148ee80b2d5d3d7a92a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af622a9190148ee80b2d5d3d7a92a20 already in progress: creating tablet
I20260317 21:19:08.398372 6666 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]). Expected version 1 got 0
I20260317 21:19:08.399372 6666 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 3 got 0
W20260317 21:19:08.403681 6666 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 consensus state has no leader
I20260317 21:19:08.418900 7467 tablet_bootstrap.cc:492] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:08.419449 7467 ts_tablet_manager.cc:1403] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.546s user 0.012s sys 0.001s
I20260317 21:19:08.423167 7467 raft_consensus.cc:359] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.431864 8106 ts_tablet_manager.cc:1916] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:08.447526 7225 debug-util.cc:398] Leaking SignalData structure 0x7b08000f5e60 after lost signal to thread 7096
I20260317 21:19:08.451421 7162 tablet_service.cc:1511] Processing CreateTablet for tablet dde40a1c77854adcabc2fff3baaf4c81 (DEFAULT_TABLE table=default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.472362 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde40a1c77854adcabc2fff3baaf4c81. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:08.472615 7225 debug-util.cc:398] Leaking SignalData structure 0x7b080011fb40 after lost signal to thread 7228
W20260317 21:19:08.457384 7152 consensus_meta.cc:287] T d69e14125d7d4412869c20759acc2336 P c0ef92fef382401a9564c17e0c223f6a: Time spent flushing consensus metadata: real 0.522s user 0.008s sys 0.000s
I20260317 21:19:08.491220 8108 raft_consensus.cc:2804] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:08.491776 8108 raft_consensus.cc:697] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:08.462970 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:08.468969 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 246) took 1882 ms. Trace:
I20260317 21:19:08.492586 8108 consensus_queue.cc:237] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.492780 7869 rpcz_store.cc:276] 0317 21:19:06.586702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.586917 (+ 215us) service_pool.cc:225] Handling call
0317 21:19:08.254436 (+1667519us) ts_tablet_manager.cc:1151] Deleting tablet 2b287c83b77548f19546927b05905746
0317 21:19:08.254483 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.335029 (+ 80546us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:08.344556 (+ 9527us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:08.468618 (+124062us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666536}
I20260317 21:19:08.494069 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 247) took 1900 ms. Trace:
I20260317 21:19:08.494362 7869 rpcz_store.cc:276] 0317 21:19:06.593483 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.593727 (+ 244us) service_pool.cc:225] Handling call
0317 21:19:08.493730 (+1900003us) ts_tablet_manager.cc:1151] Deleting tablet d4afbb1ab93a44d9ab5795b72444c690
0317 21:19:08.493774 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.494014 (+ 240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1899449}
I20260317 21:19:08.495487 7869 tablet_replica.cc:333] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
W20260317 21:19:08.495995 6971 leader_election.cc:343] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.496248 7869 raft_consensus.cc:2243] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
W20260317 21:19:08.497309 7261 leader_election.cc:343] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.497967 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet f3b5825c77ac4d69a39978932051db8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:07 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.498394 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 70560dde990843e0b7c74ae3206c58e4 with delete_type TABLET_DATA_DELETED (replaced by tablet b03b54b989d04e0990ee3ccfe7a104a4) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.500774 7869 pending_rounds.cc:70] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:19:08.501020 7869 pending_rounds.cc:77] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412494539051008 op_type: NO_OP noop_request { }
I20260317 21:19:08.504076 7869 raft_consensus.cc:2889] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:08.504442 7869 raft_consensus.cc:2272] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:08.506258 7869 ts_tablet_manager.cc:1916] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:08.514288 7467 raft_consensus.cc:385] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.514627 7467 raft_consensus.cc:740] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.516536 7467 consensus_queue.cc:260] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.499349 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:08.499779 8028 raft_consensus.cc:493] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.529919 8028 raft_consensus.cc:515] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.532008 8028 leader_election.cc:290] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.533668 7462 tablet_bootstrap.cc:654] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:08.405092 6668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4e0de322dbf141f38753722166148d81" } modify_external_catalogs: true
I20260317 21:19:08.543464 6668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_4e0de322dbf141f38753722166148d81" } modify_external_catalogs: true
I20260317 21:19:08.554353 7459 tablet_bootstrap.cc:654] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:08.558053 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:08.559142 6669 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:08.498766 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet a978fa28a31b4d589031c77526b62b8e (table default.Table_4088537e0de64ecc8cba0ebff9c50bbc [id=8a0a0d875f5e4f5fbcbb74938698fbe3]) successfully deleted
W20260317 21:19:08.573556 6662 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 74807us
W20260317 21:19:08.574355 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 35023195b9094d58827921a1dcb2ff05 because the tablet was not found. No further retry: Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05
W20260317 21:19:08.574991 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 2b287c83b77548f19546927b05905746 because the tablet was not found. No further retry: Not found: Tablet not found: 2b287c83b77548f19546927b05905746
W20260317 21:19:08.575541 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet f3b5825c77ac4d69a39978932051db8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3b5825c77ac4d69a39978932051db8c already in progress: creating tablet
W20260317 21:19:08.576176 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 70560dde990843e0b7c74ae3206c58e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70560dde990843e0b7c74ae3206c58e4 already in progress: creating tablet
I20260317 21:19:08.504905 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 2b287c83b77548f19546927b05905746 (table default.Table_e4715bec3faa471cbdcb63c57766180a [id=ff96f5df47e0414290054b29ed53479a]) successfully deleted
W20260317 21:19:08.576831 6659 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 71958.8us
W20260317 21:19:08.577373 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet d4afbb1ab93a44d9ab5795b72444c690 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d4afbb1ab93a44d9ab5795b72444c690 already in progress: creating tablet
I20260317 21:19:08.521420 6674 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_2c61fc3bab454eec8eff338bed7a87f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:08.579690 6674 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c61fc3bab454eec8eff338bed7a87f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:08.522468 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a6ed1b24bea44274bcc65ae15bec45f5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:08.523447 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_351b690e266140a3a403e918f71f9d69" } modify_external_catalogs: true
I20260317 21:19:08.580875 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_351b690e266140a3a403e918f71f9d69" } modify_external_catalogs: true
W20260317 21:19:08.581707 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6ed1b24bea44274bcc65ae15bec45f5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:08.525310 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_9c37a7d170904a5f9277a29447a1b58c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:08.583421 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c37a7d170904a5f9277a29447a1b58c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:08.526291 6671 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_05b675a3f5f649f0b419a59478fd907a" } new_table_name: "default.Table_b6c7691408304d70b7e7aee5724bcd58" modify_external_catalogs: true
W20260317 21:19:08.535290 7264 leader_election.cc:343] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.585067 7264 leader_election.cc:304] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:08.586298 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 51033.1us
I20260317 21:19:08.536255 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:08.587971 7327 raft_consensus.cc:2468] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:08.589020 8038 raft_consensus.cc:2749] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:08.591068 7303 tablet_service.cc:1511] Processing CreateTablet for tablet dde40a1c77854adcabc2fff3baaf4c81 (DEFAULT_TABLE table=default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.594213 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 43702e625cb147339bd5871a5ee83e17 (DEFAULT_TABLE table=default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.596809 7299 tablet_service.cc:1558] Processing DeleteTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.564414 8060 raft_consensus.cc:493] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.600248 8060 raft_consensus.cc:515] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.602741 7008 tablet_service.cc:1511] Processing CreateTablet for tablet dde40a1c77854adcabc2fff3baaf4c81 (DEFAULT_TABLE table=default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.602874 6973 leader_election.cc:304] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:08.603971 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde40a1c77854adcabc2fff3baaf4c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.604283 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dde40a1c77854adcabc2fff3baaf4c81. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.611219 8129 tablet.cc:1723] T 6caf05c79a5f470e8b7796d0e85e36f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:19:08.613672 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43702e625cb147339bd5871a5ee83e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.608376 7015 tablet_service.cc:1511] Processing CreateTablet for tablet 43702e625cb147339bd5871a5ee83e17 (DEFAULT_TABLE table=default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.620252 7015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43702e625cb147339bd5871a5ee83e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.627041 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:08.627336 8125 raft_consensus.cc:2804] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:08.627727 8125 raft_consensus.cc:493] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:08.627748 7327 raft_consensus.cc:2468] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:08.628072 8125 raft_consensus.cc:3060] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:08.629221 6973 leader_election.cc:304] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:08.630445 8060 leader_election.cc:290] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.632323 8060 raft_consensus.cc:2804] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:08.632928 8060 raft_consensus.cc:493] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:08.635172 8060 raft_consensus.cc:3060] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:08.639232 7467 ts_tablet_manager.cc:1434] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.219s user 0.006s sys 0.000s
I20260317 21:19:08.642522 8008 consensus_queue.cc:1048] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:08.653457 8008 consensus_queue.cc:1048] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:08.635495 7459 tablet_bootstrap.cc:492] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:08.662690 7459 ts_tablet_manager.cc:1403] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.426s user 0.009s sys 0.004s
I20260317 21:19:08.664676 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.665436 7791 tablet_replica.cc:333] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:08.666807 7791 raft_consensus.cc:2243] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:08.667600 7791 raft_consensus.cc:2272] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:08.670934 7791 ts_tablet_manager.cc:1916] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:08.677446 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 43702e625cb147339bd5871a5ee83e17 (DEFAULT_TABLE table=default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.678861 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43702e625cb147339bd5871a5ee83e17. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.679780 7467 tablet_bootstrap.cc:492] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:08.686908 7158 tablet_service.cc:1511] Processing CreateTablet for tablet d8247f6d37624ca3afcf0702eb6bdd55 (DEFAULT_TABLE table=default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.688269 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8247f6d37624ca3afcf0702eb6bdd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.691293 6668 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 829bc3959a9e4c2dbb6277cde5969b0e
I20260317 21:19:08.696939 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 829bc3959a9e4c2dbb6277cde5969b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:08.697799 6666 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_bfc5d1e1d8fa42cf920a94eab51746b3" } new_table_name: "default.Table_a24e99744e6544b696ade1dba045a68a" modify_external_catalogs: true
I20260317 21:19:08.719506 8128 tablet.cc:1723] T 6caf05c79a5f470e8b7796d0e85e36f1 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:19:08.724541 7016 tablet_service.cc:1511] Processing CreateTablet for tablet d8247f6d37624ca3afcf0702eb6bdd55 (DEFAULT_TABLE table=default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.726184 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8247f6d37624ca3afcf0702eb6bdd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.733564 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 9a1819f74297406e8ebb28ae847abdd3 (DEFAULT_TABLE table=default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.734871 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1819f74297406e8ebb28ae847abdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.746661 6669 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:08.748975 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 829bc3959a9e4c2dbb6277cde5969b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.751720 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 829bc3959a9e4c2dbb6277cde5969b0e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.756588 7305 tablet_service.cc:1511] Processing CreateTablet for tablet d8247f6d37624ca3afcf0702eb6bdd55 (DEFAULT_TABLE table=default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.757686 8038 raft_consensus.cc:493] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.758201 7083 heartbeater.cc:344] Connected to a master server at 127.6.122.124:38857
I20260317 21:19:08.758208 8038 raft_consensus.cc:515] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.758669 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d8247f6d37624ca3afcf0702eb6bdd55. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.760337 8038 leader_election.cc:290] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:08.762300 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5314c562bc9d457a8e1f4930e8d3a7ff" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:08.764474 7261 leader_election.cc:343] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.769227 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5314c562bc9d457a8e1f4930e8d3a7ff" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:08.771019 7264 leader_election.cc:343] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:19:08.771529 7264 leader_election.cc:304] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:08.772233 8038 raft_consensus.cc:2749] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:08.787900 6971 leader_election.cc:343] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:08.788978 7459 raft_consensus.cc:359] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.789798 7459 raft_consensus.cc:385] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.790104 7459 raft_consensus.cc:740] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.790992 7459 consensus_queue.cc:260] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.814879 7459 ts_tablet_manager.cc:1434] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.152s user 0.005s sys 0.000s
I20260317 21:19:08.817577 7462 tablet_bootstrap.cc:492] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:08.818122 7462 ts_tablet_manager.cc:1403] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.672s user 0.005s sys 0.008s
I20260317 21:19:08.818454 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:08.826332 7467 tablet_bootstrap.cc:700] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Time spent opening tablet: real 0.142s user 0.003s sys 0.000s
I20260317 21:19:08.828317 7467 tablet_bootstrap.cc:654] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:08.471359 6672 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 440cb8b112044a53804357056f1226f1
I20260317 21:19:08.850665 6672 catalog_manager.cc:6741] ReplaceTablet: tablet 440cb8b112044a53804357056f1226f1 deleted and replaced by tablet b35816e1de7d4b13927848538cdc3e78
I20260317 21:19:08.755059 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_e65ec8f2488d4ff99eb80d4f4f45f610"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:08.854050 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:08.571493 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58]). Expected version 3 got 0
I20260317 21:19:08.859874 6673 catalog_manager.cc:5671] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a reported cstate change: term changed from 0 to 1, leader changed from <none> to c0ef92fef382401a9564c17e0c223f6a (127.6.122.66). New cstate: current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
W20260317 21:19:08.800802 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet dde40a1c77854adcabc2fff3baaf4c81 (table default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet dde40a1c77854adcabc2fff3baaf4c81 consensus state has no leader
I20260317 21:19:08.868698 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_9a3ec369ff814877a0ff250609d2803e [id=81dc7f934d2346cd8ad83e5d718c5d7d]). Expected version 1 got 0
W20260317 21:19:08.870208 6671 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 86ae75c4ce214954beecfab9275d0ea9 consensus state has no leader
I20260317 21:19:08.875190 6670 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:08.837621 8060 raft_consensus.cc:515] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.884534 8060 leader_election.cc:290] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.889418 7459 tablet_bootstrap.cc:492] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:08.903656 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 9a1819f74297406e8ebb28ae847abdd3 (DEFAULT_TABLE table=default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.904762 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1819f74297406e8ebb28ae847abdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.905871 7302 tablet_service.cc:1558] Processing DeleteTablet for tablet 440cb8b112044a53804357056f1226f1 with delete_type TABLET_DATA_DELETED (replaced by tablet b35816e1de7d4b13927848538cdc3e78) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.906714 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet a662b365ec4846bf9e9ee243d41cb91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:08.907642 7462 raft_consensus.cc:359] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.908300 7462 raft_consensus.cc:385] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:08.908545 7462 raft_consensus.cc:740] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:08.909319 7462 consensus_queue.cc:260] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.912923 7462 ts_tablet_manager.cc:1434] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.094s user 0.008s sys 0.000s
I20260317 21:19:08.913479 8125 raft_consensus.cc:515] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.918413 7462 tablet_bootstrap.cc:492] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:08.918463 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 9a1819f74297406e8ebb28ae847abdd3 (DEFAULT_TABLE table=default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.925261 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a1819f74297406e8ebb28ae847abdd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.928879 7462 tablet_bootstrap.cc:654] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:08.950752 7298 tablet_service.cc:1511] Processing CreateTablet for tablet ff04afc5dc834ac4a51f97e29e4cc96e (DEFAULT_TABLE table=default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.951936 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff04afc5dc834ac4a51f97e29e4cc96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:08.953562 7869 ts_tablet_manager.cc:1929] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:08.954044 7869 log.cc:1199] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/78e2e20d320547b3bb1e03923de52822
I20260317 21:19:08.954452 7159 tablet_service.cc:1558] Processing DeleteTablet for tablet 440cb8b112044a53804357056f1226f1 with delete_type TABLET_DATA_DELETED (replaced by tablet b35816e1de7d4b13927848538cdc3e78) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.954341 7155 tablet_service.cc:1511] Processing CreateTablet for tablet ff04afc5dc834ac4a51f97e29e4cc96e (DEFAULT_TABLE table=default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:08.954897 7869 ts_tablet_manager.cc:1950] T 78e2e20d320547b3bb1e03923de52822 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:08.955689 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff04afc5dc834ac4a51f97e29e4cc96e. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:08.965461 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 47357.4us
I20260317 21:19:08.965034 7152 tablet_service.cc:1558] Processing DeleteTablet for tablet a662b365ec4846bf9e9ee243d41cb91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:08.969890 6675 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_74e46c4c417f4c7f95fbf9f9df94be0f" } modify_external_catalogs: true
I20260317 21:19:08.970652 6675 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_74e46c4c417f4c7f95fbf9f9df94be0f" } modify_external_catalogs: true
I20260317 21:19:08.987113 8038 raft_consensus.cc:493] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.988114 8038 raft_consensus.cc:515] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.988988 6669 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:08.990635 8038 leader_election.cc:290] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:08.993064 8125 leader_election.cc:290] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.993239 8008 raft_consensus.cc:493] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:08.993698 8008 raft_consensus.cc:515] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:08.997094 8008 leader_election.cc:290] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:08.998034 8106 ts_tablet_manager.cc:1929] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:08.998498 8106 log.cc:1199] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:08.999334 8106 ts_tablet_manager.cc:1950] T 5b066a6a5c864beb8da59bd97bacd309 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:09.002009 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 248) took 1077 ms. Trace:
I20260317 21:19:09.022346 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a662b365ec4846bf9e9ee243d41cb91a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:09.029619 8079 lock_manager.cc:585] Waited 2 seconds to obtain row lock on key '<redacted>' (key hash 9073464951982539180) tablet 00000000000000000000000000000000 cur holder 0x7b5800050700
I20260317 21:19:09.031493 7791 ts_tablet_manager.cc:1929] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:09.031961 7791 log.cc:1199] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:09.032722 7791 ts_tablet_manager.cc:1950] T 5314c562bc9d457a8e1f4930e8d3a7ff P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:09.014895 7462 tablet_bootstrap.cc:492] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:09.034860 7462 ts_tablet_manager.cc:1403] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.117s user 0.009s sys 0.026s
I20260317 21:19:09.034572 8106 rpcz_store.cc:276] 0317 21:19:07.924559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:07.924734 (+ 175us) service_pool.cc:225] Handling call
0317 21:19:08.363577 (+438843us) ts_tablet_manager.cc:1151] Deleting tablet 5b066a6a5c864beb8da59bd97bacd309
0317 21:19:08.363629 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.620186 (+256557us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:08.912377 (+292191us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:09.001917 (+ 89540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":438263}
I20260317 21:19:09.038236 7462 raft_consensus.cc:359] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.038877 7462 raft_consensus.cc:385] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.039119 7467 tablet_bootstrap.cc:492] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:09.039119 7462 raft_consensus.cc:740] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.039543 7467 ts_tablet_manager.cc:1403] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.360s user 0.009s sys 0.000s
I20260317 21:19:09.039672 7462 consensus_queue.cc:260] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.042093 7462 ts_tablet_manager.cc:1434] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
W20260317 21:19:09.024084 7261 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.038273 8106 tablet_replica.cc:333] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:09.044390 8106 raft_consensus.cc:2243] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:09.044950 8106 raft_consensus.cc:2272] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:09.051440 7459 tablet_bootstrap.cc:654] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.056212 7462 tablet_bootstrap.cc:492] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:09.061760 8106 ts_tablet_manager.cc:1916] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:09.062649 7462 tablet_bootstrap.cc:654] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.043259 7467 raft_consensus.cc:359] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.075009 7467 raft_consensus.cc:385] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.075618 7462 tablet_bootstrap.cc:492] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:09.076026 7462 ts_tablet_manager.cc:1403] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260317 21:19:09.075297 7467 raft_consensus.cc:740] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.079982 7467 consensus_queue.cc:260] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:09.081701 6666 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2fd1d8d70e9849fa9e107b2dd3355c58 consensus state has no leader
I20260317 21:19:09.083523 7467 ts_tablet_manager.cc:1434] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.044s user 0.007s sys 0.000s
I20260317 21:19:09.085680 8038 raft_consensus.cc:493] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.086212 8038 raft_consensus.cc:515] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.088321 8038 leader_election.cc:290] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.092499 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:09.089591 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d41fc13b04ec688fc38a78f55e83d" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:09.096560 7261 leader_election.cc:343] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.094664 7467 tablet_bootstrap.cc:492] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:09.098923 7459 tablet_bootstrap.cc:492] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:09.099377 7459 ts_tablet_manager.cc:1403] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.210s user 0.006s sys 0.005s
I20260317 21:19:09.078991 7462 raft_consensus.cc:359] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.102489 7462 raft_consensus.cc:385] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.103242 7462 raft_consensus.cc:740] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.103627 7459 raft_consensus.cc:359] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.104386 7459 raft_consensus.cc:385] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.107296 7459 raft_consensus.cc:740] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.107915 7459 consensus_queue.cc:260] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.112300 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 5b066a6a5c864beb8da59bd97bacd309 (table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]) successfully deleted
I20260317 21:19:09.105378 7462 consensus_queue.cc:260] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:09.113134 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 8744f6237e154dd99bc4c7e54822f7be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8744f6237e154dd99bc4c7e54822f7be already in progress: creating tablet
I20260317 21:19:09.115886 7467 tablet_bootstrap.cc:654] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.115957 7462 ts_tablet_manager.cc:1434] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.040s user 0.004s sys 0.004s
I20260317 21:19:09.118752 7459 ts_tablet_manager.cc:1434] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.019s user 0.007s sys 0.000s
I20260317 21:19:09.138787 7462 tablet_bootstrap.cc:492] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:09.140798 7459 tablet_bootstrap.cc:492] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:09.143687 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:09.145411 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 248) took 2492 ms. Trace:
I20260317 21:19:09.145762 7869 rpcz_store.cc:276] 0317 21:19:06.653497 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:06.653646 (+ 149us) service_pool.cc:225] Handling call
0317 21:19:08.495343 (+1841697us) ts_tablet_manager.cc:1151] Deleting tablet 78e2e20d320547b3bb1e03923de52822
0317 21:19:08.495381 (+ 38us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:08.721184 (+225803us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:08.953326 (+232142us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:09.145345 (+192019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1839918}
I20260317 21:19:09.147678 7462 tablet_bootstrap.cc:654] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:09.136080 8124 log.cc:927] Time spent T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: Append to log took a long time: real 0.402s user 0.030s sys 0.000s
I20260317 21:19:09.164366 7459 tablet_bootstrap.cc:654] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.170573 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b78a8f62954143d9b88c4fa88d44cbf2" } new_table_name: "default.Table_d4a5487a2b7e4218b41223f155418e27" modify_external_catalogs: true
I20260317 21:19:09.171747 7462 tablet_bootstrap.cc:492] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:09.177035 8038 raft_consensus.cc:493] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.177882 7462 ts_tablet_manager.cc:1403] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.039s user 0.013s sys 0.001s
I20260317 21:19:09.177544 8038 raft_consensus.cc:515] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.179591 8038 leader_election.cc:290] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.182499 6666 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_40785d5a99c543259bad7c3c6bcbce39" } modify_external_catalogs: true
I20260317 21:19:09.183198 6666 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_40785d5a99c543259bad7c3c6bcbce39" } modify_external_catalogs: true
I20260317 21:19:09.193832 7462 raft_consensus.cc:359] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.194555 7462 raft_consensus.cc:385] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.194831 7462 raft_consensus.cc:740] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.195413 7462 consensus_queue.cc:260] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.197458 8106 ts_tablet_manager.cc:1929] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:09.197902 8106 log.cc:1199] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/3d40549347f8463797d6791136883371
I20260317 21:19:09.198786 8106 ts_tablet_manager.cc:1950] T 3d40549347f8463797d6791136883371 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:09.199698 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 255) took 1010 ms. Trace:
I20260317 21:19:09.199893 6672 catalog_manager.cc:5671] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:09.206075 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:09.206719 6672 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:09.200093 8106 rpcz_store.cc:276] 0317 21:19:08.189471 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.190009 (+ 538us) service_pool.cc:225] Handling call
0317 21:19:09.038116 (+848107us) ts_tablet_manager.cc:1151] Deleting tablet 3d40549347f8463797d6791136883371
0317 21:19:09.038156 (+ 40us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.127769 (+ 89613us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:09.197090 (+ 69321us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:09.199630 (+ 2540us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":681257}
I20260317 21:19:09.215837 8106 tablet_replica.cc:333] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:09.216952 8106 raft_consensus.cc:2243] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:09.217484 8106 raft_consensus.cc:2272] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:09.222769 7459 tablet_bootstrap.cc:492] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:09.223239 7459 ts_tablet_manager.cc:1403] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.104s user 0.007s sys 0.003s
I20260317 21:19:09.226714 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:09.233696 7462 ts_tablet_manager.cc:1434] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.056s user 0.006s sys 0.000s
I20260317 21:19:09.234683 7462 tablet_bootstrap.cc:492] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:09.150236 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 866fdd003dd14bb9a837658e9e730f54 (DEFAULT_TABLE table=default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.248840 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866fdd003dd14bb9a837658e9e730f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.249372 8106 ts_tablet_manager.cc:1916] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:09.181321 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:09.229195 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 323717acdee841aaa1795a3ca851b40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:09.254220 7261 leader_election.cc:343] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.231555 8117 consensus_queue.cc:1048] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:09.235874 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 21e607674c734eaea207ed454097791d (DEFAULT_TABLE table=default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.257913 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e607674c734eaea207ed454097791d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.269385 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 78e2e20d320547b3bb1e03923de52822 (table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]) successfully deleted
I20260317 21:19:09.270512 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 250) took 1563 ms. Trace:
I20260317 21:19:09.270864 7869 rpcz_store.cc:276] 0317 21:19:07.707212 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:07.707346 (+ 134us) service_pool.cc:225] Handling call
0317 21:19:09.270159 (+1562813us) ts_tablet_manager.cc:1151] Deleting tablet 275e9cd7f14040d98eb85a6bb1010c2c
0317 21:19:09.270204 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.270457 (+ 253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1562244}
W20260317 21:19:09.272074 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 275e9cd7f14040d98eb85a6bb1010c2c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 275e9cd7f14040d98eb85a6bb1010c2c already in progress: creating tablet
I20260317 21:19:09.275415 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 251) took 1534 ms. Trace:
I20260317 21:19:09.275869 7869 rpcz_store.cc:276] 0317 21:19:07.741350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:07.744777 (+ 3427us) service_pool.cc:225] Handling call
0317 21:19:09.275034 (+1530257us) ts_tablet_manager.cc:1151] Deleting tablet 4af622a9190148ee80b2d5d3d7a92a20
0317 21:19:09.275080 (+ 46us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.275356 (+ 276us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1529644}
I20260317 21:19:09.278285 7462 tablet_bootstrap.cc:654] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:09.280236 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 117706us
I20260317 21:19:09.281244 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 3d40549347f8463797d6791136883371 (table default.Table_1c93ac734b4d4ff68afbd9c72d470944 [id=5939f7e719474a598b8428b8af9a5b8b]) successfully deleted
I20260317 21:19:09.282663 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a86aea973e8f4d2bb2b38b76509e80e1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:09.284336 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a86aea973e8f4d2bb2b38b76509e80e1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:09.319244 7298 tablet_service.cc:1558] Processing DeleteTablet for tablet 323717acdee841aaa1795a3ca851b40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:09.319449 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 21e607674c734eaea207ed454097791d (DEFAULT_TABLE table=default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.320106 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 866fdd003dd14bb9a837658e9e730f54 (DEFAULT_TABLE table=default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:09.322939 6971 leader_election.cc:343] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.326349 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 5314c562bc9d457a8e1f4930e8d3a7ff (table default.Table_aef4da11f840465eb7fc80e6423d7454 [id=91eadfdf7d094f9fa83003dace842023]) successfully deleted
W20260317 21:19:09.327042 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 829bc3959a9e4c2dbb6277cde5969b0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829bc3959a9e4c2dbb6277cde5969b0e already in progress: creating tablet
I20260317 21:19:09.338260 7015 tablet_service.cc:1558] Processing DeleteTablet for tablet a662b365ec4846bf9e9ee243d41cb91a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:09.341997 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 21e607674c734eaea207ed454097791d (DEFAULT_TABLE table=default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:09.342530 7006 consensus_meta.cc:287] T 9a1819f74297406e8ebb28ae847abdd3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent flushing consensus metadata: real 0.548s user 0.013s sys 0.000s
I20260317 21:19:09.341836 7459 raft_consensus.cc:359] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.343727 7459 raft_consensus.cc:385] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.344138 7459 raft_consensus.cc:740] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.344378 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e607674c734eaea207ed454097791d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.315964 7467 tablet_bootstrap.cc:492] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:09.327493 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:09.329531 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:09.348066 7467 ts_tablet_manager.cc:1403] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.255s user 0.025s sys 0.021s
I20260317 21:19:09.351246 7467 raft_consensus.cc:359] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.345428 7791 tablet_replica.cc:333] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:09.345244 7008 tablet_service.cc:1558] Processing DeleteTablet for tablet 440cb8b112044a53804357056f1226f1 with delete_type TABLET_DATA_DELETED (replaced by tablet b35816e1de7d4b13927848538cdc3e78) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:09.352236 7467 raft_consensus.cc:385] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.345821 8125 raft_consensus.cc:493] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.346889 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 866fdd003dd14bb9a837658e9e730f54 (DEFAULT_TABLE table=default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.338260 7012 tablet_service.cc:1558] Processing DeleteTablet for tablet 323717acdee841aaa1795a3ca851b40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:09.348541 7010 tablet_service.cc:1511] Processing CreateTablet for tablet ff04afc5dc834ac4a51f97e29e4cc96e (DEFAULT_TABLE table=default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.352578 7467 raft_consensus.cc:740] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.339883 7013 tablet_service.cc:1511] Processing CreateTablet for tablet f6cd3f64f9b94d089b8ecf2462007ee6 (DEFAULT_TABLE table=default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.354300 7467 consensus_queue.cc:260] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.354111 8125 raft_consensus.cc:515] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.355530 7791 raft_consensus.cc:2243] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:09.356175 7791 raft_consensus.cc:2272] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:09.357523 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 4af622a9190148ee80b2d5d3d7a92a20 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4af622a9190148ee80b2d5d3d7a92a20 already in progress: creating tablet
I20260317 21:19:09.359691 8125 leader_election.cc:290] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:09.361965 7467 ts_tablet_manager.cc:1434] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.014s user 0.002s sys 0.005s
I20260317 21:19:09.349800 7459 consensus_queue.cc:260] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.361372 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cd3f64f9b94d089b8ecf2462007ee6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:09.366566 7016 consensus_meta.cc:287] T d8247f6d37624ca3afcf0702eb6bdd55 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent flushing consensus metadata: real 0.526s user 0.013s sys 0.002s
I20260317 21:19:09.367946 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866fdd003dd14bb9a837658e9e730f54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.370185 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff04afc5dc834ac4a51f97e29e4cc96e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.371241 7791 ts_tablet_manager.cc:1916] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:09.374814 6971 leader_election.cc:343] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:09.375777 6971 leader_election.cc:343] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.383402 8038 raft_consensus.cc:493] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.383946 8038 raft_consensus.cc:515] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.385028 7459 ts_tablet_manager.cc:1434] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.161s user 0.009s sys 0.003s
I20260317 21:19:09.385582 7154 tablet_service.cc:1511] Processing CreateTablet for tablet f6cd3f64f9b94d089b8ecf2462007ee6 (DEFAULT_TABLE table=default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.385886 8038 leader_election.cc:290] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.392000 7158 tablet_service.cc:1511] Processing CreateTablet for tablet 14c394f4ea244bee8ef04f6d519ca930 (DEFAULT_TABLE table=default.Table_445e903e8c9b4d39a1d294b6f91d57ea [id=e1727a5ef7d040adb14bee89b3bb2608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.393561 7459 tablet_bootstrap.cc:492] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:09.400002 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "323717acdee841aaa1795a3ca851b40f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:09.401748 7261 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.404071 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a662b365ec4846bf9e9ee243d41cb91a" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:09.408037 7180 raft_consensus.cc:3060] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:09.408275 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cd3f64f9b94d089b8ecf2462007ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.415385 7462 tablet_bootstrap.cc:492] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:09.416049 7462 ts_tablet_manager.cc:1403] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.182s user 0.006s sys 0.006s
I20260317 21:19:09.425348 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21e607674c734eaea207ed454097791d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.425139 7298 tablet_service.cc:1511] Processing CreateTablet for tablet f6cd3f64f9b94d089b8ecf2462007ee6 (DEFAULT_TABLE table=default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.425427 6670 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_c1eb7feff9ea4b568cb56bb3761a2349" } new_table_name: "default.Table_54698fcc6a7a46deaba83055d4a1c78d" modify_external_catalogs: true
I20260317 21:19:09.428823 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_4827867befa741f69f6ab15938cfe436"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:09.429380 6671 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_cafc44dd71714a528bda6622dac7997e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:09.430506 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4827867befa741f69f6ab15938cfe436 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:09.430846 6671 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cafc44dd71714a528bda6622dac7997e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:09.424367 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_15292a1c98be431b9d3cacbc7e767c75"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:09.433059 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6cd3f64f9b94d089b8ecf2462007ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.433313 8117 raft_consensus.cc:493] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.433996 8117 raft_consensus.cc:515] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:09.434432 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_15292a1c98be431b9d3cacbc7e767c75 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:09.436690 8117 leader_election.cc:290] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:09.437173 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 866fdd003dd14bb9a837658e9e730f54. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:09.442716 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 7b7b54cbff0c4fd5813d197ddbc899cc consensus state has no leader
W20260317 21:19:09.443101 6971 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.445580 7467 tablet_bootstrap.cc:492] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:09.447167 6666 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 740ae685629041f39bbbc8ce715d662e
I20260317 21:19:09.451220 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 740ae685629041f39bbbc8ce715d662e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:09 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:09.452805 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 740ae685629041f39bbbc8ce715d662e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:09 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:09.454720 6971 consensus_peers.cc:597] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:09.456372 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c394f4ea244bee8ef04f6d519ca930. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:09.459894 7119 leader_election.cc:343] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:09.464919 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 256) took 1537 ms. Trace:
I20260317 21:19:09.468788 7462 raft_consensus.cc:359] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.474463 7869 rpcz_store.cc:276] 0317 21:19:07.927754 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:07.928192 (+ 438us) service_pool.cc:225] Handling call
0317 21:19:09.359778 (+1431586us) ts_tablet_manager.cc:1151] Deleting tablet 5b066a6a5c864beb8da59bd97bacd309
0317 21:19:09.398739 (+ 38961us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.464843 (+ 66104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1430768}
I20260317 21:19:09.475524 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 740ae685629041f39bbbc8ce715d662e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:09 UTC) from {username='slave'} at 127.0.0.1:34096
W20260317 21:19:09.478180 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 5b066a6a5c864beb8da59bd97bacd309 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5b066a6a5c864beb8da59bd97bacd309 already in progress: creating tablet
I20260317 21:19:09.479890 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 261) took 1247 ms. Trace:
I20260317 21:19:09.482841 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 14c394f4ea244bee8ef04f6d519ca930 (DEFAULT_TABLE table=default.Table_445e903e8c9b4d39a1d294b6f91d57ea [id=e1727a5ef7d040adb14bee89b3bb2608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.483996 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c394f4ea244bee8ef04f6d519ca930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.486902 8106 ts_tablet_manager.cc:1929] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:09.487409 8106 log.cc:1199] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:19:09.488209 8106 ts_tablet_manager.cc:1950] T fc8567ae10464694a1128ba2eb1d8ee1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:09.469834 7462 raft_consensus.cc:385] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.489089 7462 raft_consensus.cc:740] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.489892 7462 consensus_queue.cc:260] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.492759 7462 ts_tablet_manager.cc:1434] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.076s user 0.006s sys 0.000s
W20260317 21:19:09.502234 7410 log.cc:927] Time spent T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Append to log took a long time: real 0.188s user 0.000s sys 0.002s
I20260317 21:19:09.489795 8162 tablet.cc:1723] T 6caf05c79a5f470e8b7796d0e85e36f1 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20260317 21:19:09.525022 7869 rpcz_store.cc:276] 0317 21:19:08.232979 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.233088 (+ 109us) service_pool.cc:225] Handling call
0317 21:19:09.479483 (+1246395us) ts_tablet_manager.cc:1151] Deleting tablet 8744f6237e154dd99bc4c7e54822f7be
0317 21:19:09.479535 (+ 52us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.479827 (+ 292us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1245465}
W20260317 21:19:09.526820 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 8744f6237e154dd99bc4c7e54822f7be because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8744f6237e154dd99bc4c7e54822f7be already in progress: creating tablet
I20260317 21:19:09.527416 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 263) took 1220 ms. Trace:
I20260317 21:19:09.528952 8092 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 31726fd7f40442e49b2fe0f880793206)
I20260317 21:19:09.529851 8092 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:09.530222 8053 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31726fd7f40442e49b2fe0f880793206. Latest consensus state: current_term: 2 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:09.530887 8053 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:09.533959 8135 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:09.534969 8135 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:09.516943 8038 raft_consensus.cc:493] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.544947 8038 raft_consensus.cc:515] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.546950 8038 leader_election.cc:290] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.530673 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:09.533149 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:09.538573 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a662b365ec4846bf9e9ee243d41cb91a" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:09.547609 7328 raft_consensus.cc:3060] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:09.548030 7327 raft_consensus.cc:3060] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:09.548069 7329 raft_consensus.cc:2468] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:09.549191 7462 tablet_bootstrap.cc:492] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:09.551586 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a662b365ec4846bf9e9ee243d41cb91a" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.552428 8092 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:09.554565 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d41fc13b04ec688fc38a78f55e83d" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.557133 7869 rpcz_store.cc:276] 0317 21:19:08.306954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.308901 (+ 1947us) service_pool.cc:225] Handling call
0317 21:19:09.526779 (+1217878us) ts_tablet_manager.cc:1151] Deleting tablet f3b5825c77ac4d69a39978932051db8c
0317 21:19:09.526829 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.527327 (+ 498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1217270}
W20260317 21:19:09.559149 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet f3b5825c77ac4d69a39978932051db8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3b5825c77ac4d69a39978932051db8c already in progress: creating tablet
I20260317 21:19:09.513634 7467 tablet_bootstrap.cc:654] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.553529 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:09.563254 7261 leader_election.cc:343] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.567865 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.570091 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "323717acdee841aaa1795a3ca851b40f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.570802 7038 raft_consensus.cc:2468] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:09.572738 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 256) took 1382 ms. Trace:
I20260317 21:19:09.573127 8106 rpcz_store.cc:276] 0317 21:19:08.189838 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.190548 (+ 710us) service_pool.cc:225] Handling call
0317 21:19:09.215603 (+1025055us) ts_tablet_manager.cc:1151] Deleting tablet fc8567ae10464694a1128ba2eb1d8ee1
0317 21:19:09.215667 (+ 64us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.335748 (+120081us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:09.486417 (+150669us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:09.572665 (+ 86248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":858527}
I20260317 21:19:09.578511 6666 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_48c6d19344564ada866cf36024f05a91" } modify_external_catalogs: true
I20260317 21:19:09.579114 6666 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_48c6d19344564ada866cf36024f05a91" } modify_external_catalogs: true
I20260317 21:19:09.581027 7918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:44508 (request call id 0) took 3907 ms. Trace:
I20260317 21:19:09.583475 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet fc8567ae10464694a1128ba2eb1d8ee1 (table default.Table_1885223af9524a43b0d34db30e68e6f1 [id=1293ecb1a2c24e7e987c4c0ed2524a94]) successfully deleted
I20260317 21:19:09.500381 7459 tablet_bootstrap.cc:654] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.585196 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 14c394f4ea244bee8ef04f6d519ca930 (DEFAULT_TABLE table=default.Table_445e903e8c9b4d39a1d294b6f91d57ea [id=e1727a5ef7d040adb14bee89b3bb2608]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.586303 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14c394f4ea244bee8ef04f6d519ca930. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.595914 7328 raft_consensus.cc:2468] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:09.599992 6998 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47438 (request call id 71) took 1047 ms. Trace:
I20260317 21:19:09.600773 6998 rpcz_store.cc:276] 0317 21:19:08.552292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.552444 (+ 152us) service_pool.cc:225] Handling call
0317 21:19:08.690503 (+138059us) spinlock_profiling.cc:243] Waited 125 ms on lock 0x7b5400151b54. stack: 00007f15f787f423 00007f15f787f320 00007f15f73f4574 0000000000509544 00000000005094d9 00007f15fe8da7a1 00007f15fe96ae28 00007f15ff9ab79d 00007f15fe133104 00007f15fe133092 00007f15fe132fc1 00007f15fe132f3c 00007f15fe1321f2 00007f15fc66d34c 00007f15fc66c786 00007f15fc66c12f
0317 21:19:08.733301 (+ 42798us) spinlock_profiling.cc:243] Waited 41.1 ms on lock 0x7b54001211d4. stack: 00007f15f787f423 00007f15f787f320 00007f15f73f4574 0000000000509544 00000000005094d9 00007f15fe8da7a1 00007f15fe96ae28 00007f15ff9ab79d 00007f15fe133104 00007f15fe133092 00007f15fe132fc1 00007f15fe132f3c 00007f15fe1321f2 00007f15fc66d34c 00007f15fc66c786 00007f15fc66c12f
0317 21:19:09.338482 (+605181us) spinlock_profiling.cc:243] Waited 603 ms on lock 0x7b5400102a54. stack: 00007f15f787f423 00007f15f787f320 00007f15f73f4574 0000000000509544 00000000005094d9 00007f15fe8da7a1 00007f15fe96ae28 00007f15ff9ab79d 00007f15fe133104 00007f15fe133092 00007f15fe132fc1 00007f15fe132f3c 00007f15fe1321f2 00007f15fc66d34c 00007f15fc66c786 00007f15fc66c12f
0317 21:19:09.599792 (+261310us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":1769693824}
I20260317 21:19:09.602797 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 264) took 1264 ms. Trace:
I20260317 21:19:09.603210 7869 rpcz_store.cc:276] 0317 21:19:08.338378 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.338500 (+ 122us) service_pool.cc:225] Handling call
0317 21:19:09.567422 (+1228922us) ts_tablet_manager.cc:1151] Deleting tablet 70560dde990843e0b7c74ae3206c58e4
0317 21:19:09.602420 (+ 34998us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.602730 (+ 310us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1228367}
W20260317 21:19:09.604544 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 70560dde990843e0b7c74ae3206c58e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70560dde990843e0b7c74ae3206c58e4 already in progress: creating tablet
I20260317 21:19:09.598168 7327 raft_consensus.cc:2468] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:09.600735 7374 heartbeater.cc:499] Master 127.6.122.124:38857 was elected leader, sending a full tablet report...
I20260317 21:19:09.612463 7007 tablet_service.cc:1511] Processing CreateTablet for tablet e988430ec1074531b0e173e96983a0cb (DEFAULT_TABLE table=default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.604318 7462 tablet_bootstrap.cc:654] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.621328 6669 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]). Expected version 1 got 0
I20260317 21:19:09.624248 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:09.625018 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:09.626096 7462 tablet_bootstrap.cc:492] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:09.626484 7462 ts_tablet_manager.cc:1403] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.000s
I20260317 21:19:09.613380 7155 tablet_service.cc:1511] Processing CreateTablet for tablet e988430ec1074531b0e173e96983a0cb (DEFAULT_TABLE table=default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.628863 7462 raft_consensus.cc:359] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.629491 7462 raft_consensus.cc:385] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.629750 7462 raft_consensus.cc:740] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.630559 7462 consensus_queue.cc:260] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.633028 7462 ts_tablet_manager.cc:1434] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.006s user 0.007s sys 0.000s
I20260317 21:19:09.633461 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.634032 7462 tablet_bootstrap.cc:492] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:09.634819 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 59fce28afbeb4d8eadf63f9a97d1183d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:09.639562 7462 tablet_bootstrap.cc:654] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.641956 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e988430ec1074531b0e173e96983a0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.645203 7869 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 265) took 1295 ms. Trace:
I20260317 21:19:09.645848 7869 rpcz_store.cc:276] 0317 21:19:08.350191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.350335 (+ 144us) service_pool.cc:225] Handling call
0317 21:19:09.640555 (+1290220us) ts_tablet_manager.cc:1151] Deleting tablet 35023195b9094d58827921a1dcb2ff05
0317 21:19:09.644734 (+ 4179us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.645130 (+ 396us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289166}
I20260317 21:19:09.648921 6675 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_9a3ec369ff814877a0ff250609d2803e [id=81dc7f934d2346cd8ad83e5d718c5d7d]). Expected version 1 got 0
I20260317 21:19:09.649778 6675 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:09.651082 7462 tablet_bootstrap.cc:492] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:09.651505 7462 ts_tablet_manager.cc:1403] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.007s
W20260317 21:19:09.653139 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 35023195b9094d58827921a1dcb2ff05 because the tablet was not found. No further retry: Not found: Tablet not found: 35023195b9094d58827921a1dcb2ff05
W20260317 21:19:09.654031 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 829bc3959a9e4c2dbb6277cde5969b0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829bc3959a9e4c2dbb6277cde5969b0e already in progress: creating tablet
I20260317 21:19:09.654099 7462 raft_consensus.cc:359] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.654704 7462 raft_consensus.cc:385] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 21:19:09.654807 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 70020.8us
I20260317 21:19:09.654976 7462 raft_consensus.cc:740] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:19:09.654835 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 440cb8b112044a53804357056f1226f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440cb8b112044a53804357056f1226f1 already in progress: creating tablet
I20260317 21:19:09.655720 7462 consensus_queue.cc:260] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.656472 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fd7cf19121b5496a849118de4a29db5b" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.658228 8038 raft_consensus.cc:493] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.658697 8038 raft_consensus.cc:515] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.660696 8038 leader_election.cc:290] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.663240 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:09.664031 7264 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:19:09.664345 7462 ts_tablet_manager.cc:1434] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260317 21:19:09.664435 7264 leader_election.cc:304] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:09.665019 7261 leader_election.cc:343] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.665297 7462 tablet_bootstrap.cc:492] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:09.665495 8038 raft_consensus.cc:2749] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:09.667068 7791 ts_tablet_manager.cc:1929] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:09.667496 7791 log.cc:1199] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:09.668327 7791 ts_tablet_manager.cc:1950] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
W20260317 21:19:09.670215 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet a662b365ec4846bf9e9ee243d41cb91a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a662b365ec4846bf9e9ee243d41cb91a already in progress: creating tablet
W20260317 21:19:09.671121 7264 leader_election.cc:343] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.671527 7264 leader_election.cc:304] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:09.671934 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 323717acdee841aaa1795a3ca851b40f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 323717acdee841aaa1795a3ca851b40f already in progress: creating tablet
I20260317 21:19:09.672703 8038 raft_consensus.cc:2749] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:09.678511 7462 tablet_bootstrap.cc:654] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:09.679246 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 740ae685629041f39bbbc8ce715d662e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 740ae685629041f39bbbc8ce715d662e already in progress: creating tablet
W20260317 21:19:09.679289 7264 leader_election.cc:343] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.679669 7264 leader_election.cc:304] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:09.680747 8038 raft_consensus.cc:2749] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:09.681785 7264 leader_election.cc:304] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:09.682942 8038 raft_consensus.cc:2804] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:09.683301 8038 raft_consensus.cc:493] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:09.683759 8038 raft_consensus.cc:3060] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:09.684548 7154 tablet_service.cc:1511] Processing CreateTablet for tablet b9ded59d55424c4990d2936f92893bbe (DEFAULT_TABLE table=default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.688278 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ded59d55424c4990d2936f92893bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.689184 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e988430ec1074531b0e173e96983a0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.690969 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 258) took 1389 ms. Trace:
I20260317 21:19:09.691365 8106 rpcz_store.cc:276] 0317 21:19:08.301714 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.301892 (+ 178us) service_pool.cc:225] Handling call
0317 21:19:09.690532 (+1388640us) ts_tablet_manager.cc:1151] Deleting tablet f3b5825c77ac4d69a39978932051db8c
0317 21:19:09.690588 (+ 56us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.690902 (+ 314us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1319846}
I20260317 21:19:09.693701 8038 raft_consensus.cc:515] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.694105 6673 master_service.cc:954] ReplaceTablet: received request to replace tablet f427b15a7b2a47e18271e8884c637f4e from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:09.697818 7467 tablet_bootstrap.cc:492] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:09.698495 7467 ts_tablet_manager.cc:1403] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.253s user 0.010s sys 0.002s
I20260317 21:19:09.700284 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_c194cf09401a41589629e3c00ffbaf15" } new_table_name: "default.Table_dc73eaae587c4db2a622ecd0027888ab" modify_external_catalogs: true
I20260317 21:19:09.701352 7467 raft_consensus.cc:359] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.701975 7467 raft_consensus.cc:385] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.702212 7467 raft_consensus.cc:740] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.702906 7467 consensus_queue.cc:260] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.704771 7869 tablet_replica.cc:333] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:09.705852 7869 raft_consensus.cc:2243] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:09.706017 7459 tablet_bootstrap.cc:492] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:09.706522 7459 ts_tablet_manager.cc:1403] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.313s user 0.010s sys 0.003s
W20260317 21:19:09.709987 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet f3b5825c77ac4d69a39978932051db8c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f3b5825c77ac4d69a39978932051db8c already in progress: creating tablet
I20260317 21:19:09.709919 7459 raft_consensus.cc:359] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.711004 7459 raft_consensus.cc:385] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.711341 7459 raft_consensus.cc:740] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.711989 7459 consensus_queue.cc:260] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.711742 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet a662b365ec4846bf9e9ee243d41cb91a (table default.Table_351b690e266140a3a403e918f71f9d69 [id=ab3626e193c84f348f4cade48709a58d]) successfully deleted
I20260317 21:19:09.716856 7010 tablet_service.cc:1511] Processing CreateTablet for tablet b9ded59d55424c4990d2936f92893bbe (DEFAULT_TABLE table=default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:09.718106 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ded59d55424c4990d2936f92893bbe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:09.719677 6973 leader_election.cc:304] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:09.720553 6973 raft_consensus.cc:2686] T a662b365ec4846bf9e9ee243d41cb91a P d3cb3b49e3e94537a6e434ea4e037c3f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:19:09.721305 6973 leader_election.cc:304] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:09.729341 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 440cb8b112044a53804357056f1226f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440cb8b112044a53804357056f1226f1 already in progress: creating tablet
I20260317 21:19:09.730922 7937 consensus_queue.cc:1048] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:09.731041 7120 leader_election.cc:304] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:09.732033 8161 raft_consensus.cc:2804] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:09.732836 8008 raft_consensus.cc:2749] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:09.733203 8161 raft_consensus.cc:697] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
W20260317 21:19:09.734016 7120 leader_election.cc:343] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:09.734417 7120 leader_election.cc:304] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:09.735107 8117 raft_consensus.cc:2749] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:09.734478 8161 consensus_queue.cc:237] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.735651 8160 raft_consensus.cc:493] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.736140 8160 raft_consensus.cc:515] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.737670 7869 raft_consensus.cc:2272] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:09.738586 8160 leader_election.cc:290] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:09.740240 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e2f891e712341868d94bab1dd72a9e1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:09.741788 6971 leader_election.cc:343] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.746891 7459 ts_tablet_manager.cc:1434] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.040s user 0.003s sys 0.003s
I20260317 21:19:09.747885 7459 tablet_bootstrap.cc:492] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:09.748456 7791 tablet_replica.cc:333] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:09.749483 7791 raft_consensus.cc:2243] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:09.750134 7791 raft_consensus.cc:2272] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:09.750620 6973 leader_election.cc:304] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:09.751451 8160 raft_consensus.cc:2804] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:09.751926 8160 raft_consensus.cc:697] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:09.752167 7791 ts_tablet_manager.cc:1916] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:09.752736 8160 consensus_queue.cc:237] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.767122 7869 ts_tablet_manager.cc:1916] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:09.731119 7467 ts_tablet_manager.cc:1434] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.032s user 0.005s sys 0.000s
I20260317 21:19:09.768927 7467 tablet_bootstrap.cc:492] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:09.775681 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.778268 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "323717acdee841aaa1795a3ca851b40f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
W20260317 21:19:09.806128 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 7b7b54cbff0c4fd5813d197ddbc899cc (table default.Table_d4a5487a2b7e4218b41223f155418e27 [id=b816766f31714663b685cfcc9a9f537a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 7b7b54cbff0c4fd5813d197ddbc899cc consensus state has no leader
I20260317 21:19:09.826702 8008 raft_consensus.cc:493] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:09.827556 8008 raft_consensus.cc:515] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.829264 7459 tablet_bootstrap.cc:654] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.831103 8008 leader_election.cc:290] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:09.831602 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:09.851888 7467 tablet_bootstrap.cc:654] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.854912 7459 tablet_bootstrap.cc:492] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:09.860184 7459 ts_tablet_manager.cc:1403] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.112s user 0.013s sys 0.000s
I20260317 21:19:09.864437 7459 raft_consensus.cc:359] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.874995 7459 raft_consensus.cc:385] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.876931 7459 raft_consensus.cc:740] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.887133 7459 consensus_queue.cc:260] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.711503 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 259) took 1371 ms. Trace:
I20260317 21:19:09.898011 8106 rpcz_store.cc:276] 0317 21:19:08.339697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.340031 (+ 334us) service_pool.cc:225] Handling call
0317 21:19:09.692356 (+1352325us) ts_tablet_manager.cc:1151] Deleting tablet 70560dde990843e0b7c74ae3206c58e4
0317 21:19:09.692401 (+ 45us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:09.711431 (+ 19030us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1321460}
I20260317 21:19:09.899134 8106 tablet_replica.cc:333] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:09.900060 8106 raft_consensus.cc:2243] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:09.900568 8106 raft_consensus.cc:2272] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:09.902174 8038 leader_election.cc:290] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.902452 8106 ts_tablet_manager.cc:1916] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:09.908402 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "323717acdee841aaa1795a3ca851b40f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:09.909691 7869 ts_tablet_manager.cc:1929] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:09.912720 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:09.913611 7869 log.cc:1199] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/59fce28afbeb4d8eadf63f9a97d1183d
I20260317 21:19:09.914536 7869 ts_tablet_manager.cc:1950] T 59fce28afbeb4d8eadf63f9a97d1183d P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:09.917416 7459 ts_tablet_manager.cc:1434] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.055s user 0.006s sys 0.000s
I20260317 21:19:09.918417 7459 tablet_bootstrap.cc:492] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:09.919384 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 59fce28afbeb4d8eadf63f9a97d1183d (table default.Table_6d4e07837b8945f981be0ae07f5b4ea0 [id=2e0f6bf62c664a6cb3f2bdad9a74e11b]) successfully deleted
W20260317 21:19:09.925601 7261 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:09.928053 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:09.939401 6973 leader_election.cc:343] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:09.939992 6973 leader_election.cc:304] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:09.941818 8161 raft_consensus.cc:2749] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:09.942477 8161 consensus_queue.cc:1048] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:09.947307 7791 ts_tablet_manager.cc:1929] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:09.947721 7791 log.cc:1199] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/323717acdee841aaa1795a3ca851b40f
I20260317 21:19:09.957794 7791 ts_tablet_manager.cc:1950] T 323717acdee841aaa1795a3ca851b40f P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:09.960234 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 323717acdee841aaa1795a3ca851b40f (table default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]) successfully deleted
I20260317 21:19:09.960436 7890 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" instance_seqno: 1773782324628903) as {username='slave'} at 127.6.122.66:36101; Asking this server to re-register.
I20260317 21:19:09.955135 7459 tablet_bootstrap.cc:654] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:09.965355 7467 tablet_bootstrap.cc:492] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:09.965773 7467 ts_tablet_manager.cc:1403] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.197s user 0.008s sys 0.008s
W20260317 21:19:09.973742 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 259703us
I20260317 21:19:09.968228 7467 raft_consensus.cc:359] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.978955 7467 raft_consensus.cc:385] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:09.979251 7467 raft_consensus.cc:740] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:09.986915 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:09.987968 8092 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 21:19:09.993854 6670 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e06a1488c60f4d1387417ce30579fbf4 consensus state has no leader
I20260317 21:19:10.002188 7229 heartbeater.cc:461] Registering TS with master...
I20260317 21:19:10.006855 7895 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" instance_seqno: 1773782321930335) as {username='slave'} at 127.6.122.65:54487; Asking this server to re-register.
I20260317 21:19:10.008961 6671 master_service.cc:954] ReplaceTablet: received request to replace tablet 646a115e133b43eebc1674a862ab7838 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:10.010573 7083 heartbeater.cc:461] Registering TS with master...
W20260317 21:19:10.023532 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 740ae685629041f39bbbc8ce715d662e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 740ae685629041f39bbbc8ce715d662e already in progress: creating tablet
I20260317 21:19:10.023640 7895 ts_manager.cc:194] Registered new tserver with Master: c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:09.985030 7467 consensus_queue.cc:260] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.030581 8038 raft_consensus.cc:493] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.031109 8038 raft_consensus.cc:515] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.035215 8106 ts_tablet_manager.cc:1929] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:10.035647 8106 log.cc:1199] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:10.035868 6666 catalog_manager.cc:5953] Not sending DeleteTablet requests; no consensus state for tablet 325059bee9e243dd98abb9bf9621b7db
I20260317 21:19:10.036331 8106 ts_tablet_manager.cc:1950] T 5314c562bc9d457a8e1f4930e8d3a7ff P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
W20260317 21:19:10.043148 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 70560dde990843e0b7c74ae3206c58e4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70560dde990843e0b7c74ae3206c58e4 already in progress: creating tablet
I20260317 21:19:10.061256 6672 catalog_manager.cc:6216] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58] alter complete (version 3)
I20260317 21:19:10.062824 7467 ts_tablet_manager.cc:1434] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.097s user 0.007s sys 0.000s
I20260317 21:19:10.063721 7467 tablet_bootstrap.cc:492] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:10.065145 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 262) took 1468 ms. Trace:
I20260317 21:19:10.065495 8106 rpcz_store.cc:276] 0317 21:19:08.596363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.596549 (+ 186us) service_pool.cc:225] Handling call
0317 21:19:09.898972 (+1302423us) ts_tablet_manager.cc:1151] Deleting tablet 5314c562bc9d457a8e1f4930e8d3a7ff
0317 21:19:09.899020 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:10.022671 (+123651us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:10.034971 (+ 12300us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:10.065079 (+ 30108us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1301416}
I20260317 21:19:10.066989 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 263) took 1315 ms. Trace:
I20260317 21:19:10.067301 8106 rpcz_store.cc:276] 0317 21:19:08.751445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.751590 (+ 145us) service_pool.cc:225] Handling call
0317 21:19:10.066641 (+1315051us) ts_tablet_manager.cc:1151] Deleting tablet 829bc3959a9e4c2dbb6277cde5969b0e
0317 21:19:10.066682 (+ 41us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:10.066934 (+ 252us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1314390}
I20260317 21:19:10.068148 6667 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f9ab9898a12b4ffeb9a2c721ee8f05c3" } modify_external_catalogs: true
I20260317 21:19:10.069702 7462 tablet_bootstrap.cc:492] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:10.070197 7462 ts_tablet_manager.cc:1403] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.405s user 0.017s sys 0.001s
I20260317 21:19:10.069955 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_2cd62a2d2daa4868a58114c426bca181"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:10.071491 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cd62a2d2daa4868a58114c426bca181 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.073577 7462 raft_consensus.cc:359] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.074239 7462 raft_consensus.cc:385] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.074479 7462 raft_consensus.cc:740] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.083037 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet a978fa28a31b4d589031c77526b62b8e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:10.083781 7869 tablet_replica.cc:333] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:10.084502 7869 raft_consensus.cc:2243] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:10.084981 7869 raft_consensus.cc:2272] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:10.086547 7869 ts_tablet_manager.cc:1916] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:10.076611 6672 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 78) took 1214 ms (client timeout 1000 ms). Trace:
W20260317 21:19:10.080040 6669 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 60) took 1002 ms (client timeout 1000 ms). Trace:
W20260317 21:19:10.091425 6669 rpcz_store.cc:269] 0317 21:19:09.078023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:09.078463 (+ 440us) service_pool.cc:225] Handling call
0317 21:19:09.173187 (+ 94724us) spinlock_profiling.cc:243] Waited 93.1 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:10.079967 (+906780us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":2,"spinlock_wait_cycles":214023552}
I20260317 21:19:10.093847 8038 leader_election.cc:290] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:10.094457 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "039d8bed256a43a6b98075b8430cbf3b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:10.083245 6667 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_f9ab9898a12b4ffeb9a2c721ee8f05c3" } modify_external_catalogs: true
W20260317 21:19:10.095768 7261 leader_election.cc:343] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:10.090464 6672 rpcz_store.cc:269] 0317 21:19:08.862571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.864393 (+ 1822us) service_pool.cc:225] Handling call
0317 21:19:08.932473 (+ 68080us) spinlock_profiling.cc:243] Waited 65.7 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:10.076520 (+1144047us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:09.304322 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:09.304766 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:09.304797 (+ 31us) write_op.cc:435] Acquired schema lock
0317 21:19:09.304810 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:09.305069 (+ 259us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:09.305106 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:09.305127 (+ 21us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:09.305266 (+ 139us) tablet.cc:700] Row locks acquired
0317 21:19:09.305280 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:09.305387 (+ 107us) write_op.cc:270] Start()
0317 21:19:09.305473 (+ 86us) write_op.cc:276] Timestamp: P: 1773782349305344 usec, L: 0
0317 21:19:09.305492 (+ 19us) op_driver.cc:348] REPLICATION: starting
0317 21:19:09.306054 (+ 562us) log.cc:844] Serialized 530 byte log entry
0317 21:19:09.614044 (+307990us) op_driver.cc:464] REPLICATION: finished
0317 21:19:09.614603 (+ 559us) write_op.cc:301] APPLY: starting
0317 21:19:09.614659 (+ 56us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:09.614877 (+ 218us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:09.614968 (+ 91us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:09.615139 (+ 171us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:09.615269 (+ 130us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:09.615310 (+ 41us) write_op.cc:312] APPLY: finished
0317 21:19:09.615494 (+ 184us) log.cc:844] Serialized 23 byte log entry
0317 21:19:09.615679 (+ 185us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:09.615742 (+ 63us) write_op.cc:454] Released schema lock
0317 21:19:09.615773 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:09.615994 (+ 221us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:09.616018 (+ 24us) write_op.cc:454] Released schema lock
Related trace 'op':
0317 21:19:09.620682 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:09.621119 (+ 437us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:09.621143 (+ 24us) write_op.cc:435] Acquired schema lock
0317 21:19:09.621155 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:09.621455 (+ 300us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:09.621492 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:09.621515 (+ 23us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:09.621657 (+ 142us) tablet.cc:700] Row locks acquired
0317 21:19:09.621669 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:09.621773 (+ 104us) write_op.cc:270] Start()
0317 21:19:09.621858 (+ 85us) write_op.cc:276] Timestamp: P: 1773782349621702 usec, L: 0
0317 21:19:09.621873 (+ 15us) op_driver.cc:348] REPLICATION: starting
0317 21:19:09.622347 (+ 474us) log.cc:844] Serialized 356 byte log entry
0317 21:19:10.052544 (+430197us) op_driver.cc:464] REPLICATION: finished
0317 21:19:10.059096 (+ 6552us) write_op.cc:301] APPLY: starting
0317 21:19:10.059151 (+ 55us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:10.059346 (+ 195us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:10.059427 (+ 81us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:10.059709 (+ 282us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:10.059842 (+ 133us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:10.059889 (+ 47us) write_op.cc:312] APPLY: finished
0317 21:19:10.060071 (+ 182us) log.cc:844] Serialized 23 byte log entry
0317 21:19:10.060737 (+ 666us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:10.060797 (+ 60us) write_op.cc:454] Released schema lock
0317 21:19:10.060825 (+ 28us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:10.061028 (+ 203us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:10.061056 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":3,"spinlock_wait_cycles":151072000,"child_traces":[["op",{"apply.queue_time_us":392,"apply.run_cpu_time_us":1358,"apply.run_wall_time_us":1358,"num_ops":1,"prepare.queue_time_us":96006,"prepare.run_cpu_time_us":2033,"prepare.run_wall_time_us":6276,"replication_time_us":308450}],["op",{"apply.queue_time_us":6372,"apply.run_cpu_time_us":1250,"apply.run_wall_time_us":1916,"num_ops":1,"prepare.queue_time_us":117,"prepare.run_cpu_time_us":1889,"prepare.run_wall_time_us":1897,"replication_time_us":430584}]]}
I20260317 21:19:10.101687 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 265) took 1196 ms. Trace:
I20260317 21:19:10.102092 8106 rpcz_store.cc:276] 0317 21:19:08.905570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.905711 (+ 141us) service_pool.cc:225] Handling call
0317 21:19:10.068204 (+1162493us) ts_tablet_manager.cc:1151] Deleting tablet 440cb8b112044a53804357056f1226f1
0317 21:19:10.068243 (+ 39us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:10.101607 (+ 33364us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1161874}
I20260317 21:19:10.103178 8106 tablet_replica.cc:333] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:10.103834 6675 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:10.104281 8164 catalog_manager.cc:1269] Loaded cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:19:10.104001 8106 raft_consensus.cc:2243] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:10.105710 8106 raft_consensus.cc:2272] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:10.106747 7459 tablet_bootstrap.cc:492] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:10.107167 7459 ts_tablet_manager.cc:1403] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.189s user 0.011s sys 0.002s
I20260317 21:19:10.109714 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 275e9cd7f14040d98eb85a6bb1010c2c with delete_type TABLET_DATA_DELETED (replaced by tablet 76bd0f5369cf400299039e4631ed11b0) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:10.109889 7459 raft_consensus.cc:359] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.110479 7459 raft_consensus.cc:385] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.110723 7459 raft_consensus.cc:740] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.111289 7459 consensus_queue.cc:260] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.113373 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "63a27911e0e545f5811eb075997d0b0f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:10.115736 7869 ts_tablet_manager.cc:1929] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:10.116168 7869 log.cc:1199] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/a978fa28a31b4d589031c77526b62b8e
I20260317 21:19:10.115983 7305 tablet_service.cc:1511] Processing CreateTablet for tablet e988430ec1074531b0e173e96983a0cb (DEFAULT_TABLE table=default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:10.116811 7869 ts_tablet_manager.cc:1950] T a978fa28a31b4d589031c77526b62b8e P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:10.117013 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e988430ec1074531b0e173e96983a0cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:10.119598 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet a978fa28a31b4d589031c77526b62b8e (table default.Table_4088537e0de64ecc8cba0ebff9c50bbc [id=8a0a0d875f5e4f5fbcbb74938698fbe3]) successfully deleted
I20260317 21:19:10.122795 7869 tablet_replica.cc:333] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:10.124886 8164 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: loading cluster ID for follower catalog manager: success
I20260317 21:19:10.125907 7869 raft_consensus.cc:2243] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:10.126389 7869 raft_consensus.cc:2272] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:10.125524 7462 consensus_queue.cc:260] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.127038 7467 tablet_bootstrap.cc:654] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.129518 7462 ts_tablet_manager.cc:1434] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.059s user 0.002s sys 0.005s
I20260317 21:19:10.130417 7462 tablet_bootstrap.cc:492] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:10.129498 7869 ts_tablet_manager.cc:1916] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:10.131497 7459 ts_tablet_manager.cc:1434] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260317 21:19:10.137735 7459 tablet_bootstrap.cc:492] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:10.137518 7299 tablet_service.cc:1511] Processing CreateTablet for tablet b9ded59d55424c4990d2936f92893bbe (DEFAULT_TABLE table=default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:10.138635 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b9ded59d55424c4990d2936f92893bbe. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:10.148983 7264 leader_election.cc:343] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.149413 7264 leader_election.cc:304] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:10.167762 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 84fcdc682c2f4bcdb61ec832ec42ea1b has no consensus state
I20260317 21:19:10.170962 7869 ts_tablet_manager.cc:1929] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:10.171314 7869 log.cc:1199] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/275e9cd7f14040d98eb85a6bb1010c2c
I20260317 21:19:10.171932 7869 ts_tablet_manager.cc:1950] T 275e9cd7f14040d98eb85a6bb1010c2c P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:10.173228 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 275e9cd7f14040d98eb85a6bb1010c2c (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]) successfully deleted
I20260317 21:19:10.174027 8164 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: acquiring CA information for follower catalog manager: success
I20260317 21:19:10.174791 6673 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet f427b15a7b2a47e18271e8884c637f4e
I20260317 21:19:10.176191 6673 catalog_manager.cc:6741] ReplaceTablet: tablet f427b15a7b2a47e18271e8884c637f4e deleted and replaced by tablet 8a2df0b2a73f4ed4b349efbb4d424afa
I20260317 21:19:10.177557 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet f427b15a7b2a47e18271e8884c637f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 8a2df0b2a73f4ed4b349efbb4d424afa) from {username='slave'} at 127.0.0.1:42830
W20260317 21:19:10.179136 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet f427b15a7b2a47e18271e8884c637f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f427b15a7b2a47e18271e8884c637f4e already in progress: creating tablet
I20260317 21:19:10.183573 7459 tablet_bootstrap.cc:654] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.185899 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet f427b15a7b2a47e18271e8884c637f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 8a2df0b2a73f4ed4b349efbb4d424afa) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:10.187592 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet f427b15a7b2a47e18271e8884c637f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f427b15a7b2a47e18271e8884c637f4e already in progress: creating tablet
I20260317 21:19:10.187626 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:10.189275 6673 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_9a3ec369ff814877a0ff250609d2803e [id=81dc7f934d2346cd8ad83e5d718c5d7d]). Expected version 1 got 0
W20260317 21:19:10.189452 7264 leader_election.cc:343] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.190048 7264 leader_election.cc:304] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.199661 7467 tablet_bootstrap.cc:492] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:10.200187 7467 ts_tablet_manager.cc:1403] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.137s user 0.009s sys 0.003s
I20260317 21:19:10.202907 7467 raft_consensus.cc:359] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.207222 8106 ts_tablet_manager.cc:1916] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:10.209100 8038 raft_consensus.cc:2749] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.209898 8173 raft_consensus.cc:2749] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.210197 6674 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_9a3ec369ff814877a0ff250609d2803e" } new_table_name: "default.Table_154f9dc4f8214c7b832788cb604a6f70" modify_external_catalogs: true
I20260317 21:19:10.213311 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:10.219694 7467 raft_consensus.cc:385] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.219441 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_6254d435474446bca35e4c78478b186f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:10.219980 7467 raft_consensus.cc:740] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.220551 7467 consensus_queue.cc:260] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.234308 7467 ts_tablet_manager.cc:1434] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.034s user 0.006s sys 0.000s
W20260317 21:19:10.239665 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6254d435474446bca35e4c78478b186f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.245369 7459 tablet_bootstrap.cc:492] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:10.245790 7459 ts_tablet_manager.cc:1403] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.108s user 0.000s sys 0.016s
I20260317 21:19:10.246259 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_9967a6acfcf84b2c9759c612f25702e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:10.248343 7459 raft_consensus.cc:359] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.255097 7459 raft_consensus.cc:385] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.255427 7459 raft_consensus.cc:740] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.256069 7459 consensus_queue.cc:260] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.259331 7462 tablet_bootstrap.cc:654] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.263181 7467 tablet_bootstrap.cc:492] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:19:10.267205 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 77782.5us
W20260317 21:19:10.268010 7264 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:19:10.268572 7264 leader_election.cc:304] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.270391 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 5314c562bc9d457a8e1f4930e8d3a7ff (table default.Table_aef4da11f840465eb7fc80e6423d7454 [id=91eadfdf7d094f9fa83003dace842023]) successfully deleted
W20260317 21:19:10.271425 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 829bc3959a9e4c2dbb6277cde5969b0e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 829bc3959a9e4c2dbb6277cde5969b0e already in progress: creating tablet
W20260317 21:19:10.273028 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 440cb8b112044a53804357056f1226f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 440cb8b112044a53804357056f1226f1 already in progress: creating tablet
I20260317 21:19:10.272264 8192 raft_consensus.cc:2749] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.275851 7119 leader_election.cc:304] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.277539 6668 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_445e903e8c9b4d39a1d294b6f91d57ea" } modify_external_catalogs: true
I20260317 21:19:10.277876 8008 raft_consensus.cc:493] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.278090 6668 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_445e903e8c9b4d39a1d294b6f91d57ea" } modify_external_catalogs: true
I20260317 21:19:10.278437 8008 raft_consensus.cc:515] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:10.279867 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9967a6acfcf84b2c9759c612f25702e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.280755 7299 tablet_service.cc:1558] Processing DeleteTablet for tablet f427b15a7b2a47e18271e8884c637f4e with delete_type TABLET_DATA_DELETED (replaced by tablet 8a2df0b2a73f4ed4b349efbb4d424afa) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:10.281376 8117 raft_consensus.cc:2749] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:10.283025 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.271291 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "039d8bed256a43a6b98075b8430cbf3b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:10.283661 8106 ts_tablet_manager.cc:1929] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:10.282590 8008 leader_election.cc:290] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:10.285226 7459 ts_tablet_manager.cc:1434] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.039s user 0.006s sys 0.000s
I20260317 21:19:10.288919 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:10.291249 7459 tablet_bootstrap.cc:492] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:10.296145 8106 log.cc:1199] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:10.297500 8106 ts_tablet_manager.cc:1950] T a662b365ec4846bf9e9ee243d41cb91a P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:10.300323 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 266) took 1394 ms. Trace:
I20260317 21:19:10.311966 7459 tablet_bootstrap.cc:654] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.320775 7467 tablet_bootstrap.cc:654] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.320571 8106 rpcz_store.cc:276] 0317 21:19:08.905985 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:08.906590 (+ 605us) service_pool.cc:225] Handling call
0317 21:19:10.103013 (+1196423us) ts_tablet_manager.cc:1151] Deleting tablet a662b365ec4846bf9e9ee243d41cb91a
0317 21:19:10.103057 (+ 44us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:10.261338 (+158281us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:10.283407 (+ 22069us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:10.300257 (+ 16850us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1195881}
I20260317 21:19:10.323050 8106 tablet_replica.cc:333] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:10.327651 8106 raft_consensus.cc:2243] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:10.333058 8106 raft_consensus.cc:2272] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:10.338979 7892 ts_manager.cc:194] Registered new tserver with Master: d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:10.340195 8106 ts_tablet_manager.cc:1916] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:10.343865 7459 tablet_bootstrap.cc:492] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:10.348809 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet a662b365ec4846bf9e9ee243d41cb91a (table default.Table_351b690e266140a3a403e918f71f9d69 [id=ab3626e193c84f348f4cade48709a58d]) successfully deleted
I20260317 21:19:10.350026 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "221c71b908f942ca97f111f857f0335a" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:10.351490 7467 tablet_bootstrap.cc:492] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:10.351657 7328 raft_consensus.cc:2393] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:10.351886 7467 ts_tablet_manager.cc:1403] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.090s user 0.013s sys 0.002s
I20260317 21:19:10.354765 7467 raft_consensus.cc:359] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.355419 7467 raft_consensus.cc:385] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.355753 7467 raft_consensus.cc:740] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
W20260317 21:19:10.358114 7264 leader_election.cc:343] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.358625 7264 leader_election.cc:304] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.361670 7459 ts_tablet_manager.cc:1403] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.071s user 0.008s sys 0.004s
I20260317 21:19:10.356400 7467 consensus_queue.cc:260] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.368414 8192 raft_consensus.cc:2749] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.364172 7459 raft_consensus.cc:359] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.376890 7459 raft_consensus.cc:385] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.377265 7459 raft_consensus.cc:740] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.385349 6672 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_b6c7691408304d70b7e7aee5724bcd58" } new_table_name: "default.Table_15fe620085e9499295d76e3a7b3df064" modify_external_catalogs: true
I20260317 21:19:10.385232 7459 consensus_queue.cc:260] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.387295 7459 ts_tablet_manager.cc:1434] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.025s user 0.004s sys 0.000s
I20260317 21:19:10.391949 7119 leader_election.cc:304] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.393108 8194 raft_consensus.cc:3060] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:10.379290 7467 ts_tablet_manager.cc:1434] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.027s user 0.008s sys 0.000s
I20260317 21:19:10.402245 7467 tablet_bootstrap.cc:492] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:10.416939 8192 raft_consensus.cc:493] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.417426 8192 raft_consensus.cc:515] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.419435 8192 leader_election.cc:290] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:10.422858 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:10.424103 7467 tablet_bootstrap.cc:654] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:10.425788 7261 leader_election.cc:343] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.436501 7459 tablet_bootstrap.cc:492] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:10.444591 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:10.445291 7038 raft_consensus.cc:2468] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:10.450835 7181 raft_consensus.cc:3060] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:10.457181 7264 leader_election.cc:304] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:10.467087 8127 raft_consensus.cc:493] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.467736 8127 raft_consensus.cc:515] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.471685 8127 leader_election.cc:290] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:10.482658 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_1342223726fb480aba4932d06e045aaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:10.485488 6973 leader_election.cc:343] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:19:10.492597 7462 tablet_bootstrap.cc:492] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:10.498126 7462 ts_tablet_manager.cc:1403] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.368s user 0.013s sys 0.022s
I20260317 21:19:10.497908 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd99ca9f6c540e48e29d8659fbfe777" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:10.499379 6971 leader_election.cc:343] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.499866 6971 leader_election.cc:304] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:10.501080 8127 raft_consensus.cc:2749] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.506304 7459 tablet_bootstrap.cc:654] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.517220 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_147e7f28d4344faf9b6289f6c999b6a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:10.527273 8106 ts_tablet_manager.cc:1929] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:10.529179 7895 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" instance_seqno: 1773782326925162) as {username='slave'} at 127.6.122.67:56885; Asking this server to re-register.
I20260317 21:19:10.532132 8106 log.cc:1199] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/323717acdee841aaa1795a3ca851b40f
I20260317 21:19:10.532903 8106 ts_tablet_manager.cc:1950] T 323717acdee841aaa1795a3ca851b40f P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:10.536763 7459 tablet_bootstrap.cc:492] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:10.537209 7459 ts_tablet_manager.cc:1403] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.101s user 0.010s sys 0.001s
I20260317 21:19:10.537727 7467 tablet_bootstrap.cc:492] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:10.538748 8192 raft_consensus.cc:2804] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:10.539111 8192 raft_consensus.cc:493] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:10.539431 8192 raft_consensus.cc:3060] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:10.539593 7459 raft_consensus.cc:359] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.540302 7459 raft_consensus.cc:385] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.542281 7467 ts_tablet_manager.cc:1403] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.140s user 0.014s sys 0.022s
I20260317 21:19:10.547654 8192 raft_consensus.cc:515] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.551085 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:10.552889 7261 leader_election.cc:343] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.554719 8195 raft_consensus.cc:493] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.555213 8195 raft_consensus.cc:515] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.557381 8195 leader_election.cc:290] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:10.560092 7120 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:10.554059 7462 raft_consensus.cc:359] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.561151 7462 raft_consensus.cc:385] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.540561 7459 raft_consensus.cc:740] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.561482 7462 raft_consensus.cc:740] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.562215 7462 consensus_queue.cc:260] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.565150 7459 consensus_queue.cc:260] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.580915 7459 ts_tablet_manager.cc:1434] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.043s user 0.006s sys 0.000s
I20260317 21:19:10.585366 7467 raft_consensus.cc:359] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.590903 7467 raft_consensus.cc:385] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.600970 8208 tablet.cc:1723] T 564a02888d854ea58d7a590cba816b86 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:19:10.597889 7467 raft_consensus.cc:740] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.613735 8209 tablet.cc:1723] T 564a02888d854ea58d7a590cba816b86 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:19:10.612521 7459 tablet_bootstrap.cc:492] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:10.537860 8164 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260317 21:19:10.625382 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1342223726fb480aba4932d06e045aaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:10.629240 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_147e7f28d4344faf9b6289f6c999b6a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.639689 8127 consensus_queue.cc:1048] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:10.608335 7467 consensus_queue.cc:260] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.662382 7467 ts_tablet_manager.cc:1434] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.116s user 0.006s sys 0.003s
I20260317 21:19:10.662421 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_a4ca7b6127064c19b73aee408bc49282"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:10.664002 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a4ca7b6127064c19b73aee408bc49282 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.654635 7459 tablet_bootstrap.cc:654] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.680918 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_7864164e6f474a35a89094d36d4254ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:10.682523 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7864164e6f474a35a89094d36d4254ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.686677 8127 consensus_queue.cc:1048] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:10.697636 6675 master_service.cc:954] ReplaceTablet: received request to replace tablet 7eb432b6bccf4e32a2fa9d4979586c52 from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:10.706135 8127 raft_consensus.cc:493] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.706599 8127 raft_consensus.cc:515] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.712986 8127 leader_election.cc:290] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:10.721716 7459 tablet_bootstrap.cc:492] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:10.722199 7459 ts_tablet_manager.cc:1403] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.110s user 0.010s sys 0.001s
I20260317 21:19:10.724471 7459 raft_consensus.cc:359] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.725147 7459 raft_consensus.cc:385] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.725404 7459 raft_consensus.cc:740] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.741258 6675 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:10.742676 6675 catalog_manager.cc:6741] ReplaceTablet: tablet 7eb432b6bccf4e32a2fa9d4979586c52 deleted and replaced by tablet 6deece45839849088dad3d873cc925b2
I20260317 21:19:10.745234 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 with delete_type TABLET_DATA_DELETED (replaced by tablet 6deece45839849088dad3d873cc925b2) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:10.746189 7154 tablet_service.cc:1558] Processing DeleteTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 with delete_type TABLET_DATA_DELETED (replaced by tablet 6deece45839849088dad3d873cc925b2) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:10.745798 7459 consensus_queue.cc:260] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.748600 7459 ts_tablet_manager.cc:1434] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.026s user 0.003s sys 0.003s
I20260317 21:19:10.753310 7459 tablet_bootstrap.cc:492] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:10.568044 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 110982us
I20260317 21:19:10.762936 7459 tablet_bootstrap.cc:654] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.629419 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 269) took 1311 ms. Trace:
I20260317 21:19:10.763898 7299 tablet_service.cc:1558] Processing DeleteTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 with delete_type TABLET_DATA_DELETED (replaced by tablet 6deece45839849088dad3d873cc925b2) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:10.763662 8106 rpcz_store.cc:276] 0317 21:19:09.317729 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:09.319125 (+ 1396us) service_pool.cc:225] Handling call
0317 21:19:10.322876 (+1003751us) ts_tablet_manager.cc:1151] Deleting tablet 323717acdee841aaa1795a3ca851b40f
0317 21:19:10.322926 (+ 50us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:10.396096 (+ 73170us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:10.521587 (+125491us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:10.629335 (+107748us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86487,"tablet-delete.queue_time_us":1002922}
I20260317 21:19:10.765317 8106 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34096 (request call id 273) took 1290 ms. Trace:
I20260317 21:19:10.765304 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 323717acdee841aaa1795a3ca851b40f (table default.Table_74e46c4c417f4c7f95fbf9f9df94be0f [id=558216b9c0c440e2819f192e6790bc8e]) successfully deleted
I20260317 21:19:10.765642 8106 rpcz_store.cc:276] 0317 21:19:09.475202 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:09.475344 (+ 142us) service_pool.cc:225] Handling call
0317 21:19:10.764938 (+1289594us) ts_tablet_manager.cc:1151] Deleting tablet 740ae685629041f39bbbc8ce715d662e
0317 21:19:10.764985 (+ 47us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:10.765260 (+ 275us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1288932}
W20260317 21:19:10.766050 7119 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:10.766460 7119 leader_election.cc:304] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.767272 8106 tablet_replica.cc:333] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:10.639983 8186 raft_consensus.cc:493] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.768047 8106 raft_consensus.cc:2243] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:10.767913 8186 raft_consensus.cc:515] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.768823 8195 raft_consensus.cc:2749] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.768530 8106 raft_consensus.cc:2272] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:10.769151 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 740ae685629041f39bbbc8ce715d662e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 740ae685629041f39bbbc8ce715d662e already in progress: creating tablet
I20260317 21:19:10.713343 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:10.769840 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet f427b15a7b2a47e18271e8884c637f4e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f427b15a7b2a47e18271e8884c637f4e already in progress: creating tablet
I20260317 21:19:10.770010 7326 raft_consensus.cc:2468] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:10.771180 8106 ts_tablet_manager.cc:1916] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:10.772588 8186 leader_election.cc:290] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:10.773886 6973 leader_election.cc:304] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:10.774636 8160 raft_consensus.cc:2804] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:10.774987 8160 raft_consensus.cc:493] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:10.775321 8160 raft_consensus.cc:3060] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:10.775271 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:10.775843 7038 raft_consensus.cc:3060] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:10.760907 8192 leader_election.cc:290] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:10.781250 7467 tablet_bootstrap.cc:492] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:10.781932 8210 tablet.cc:1723] T 564a02888d854ea58d7a590cba816b86 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:19:10.784662 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:10.784564 8160 raft_consensus.cc:515] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.786243 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:10.787374 7261 leader_election.cc:343] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.787336 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:10.788113 7038 raft_consensus.cc:2468] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:10.788087 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:10.788625 6971 leader_election.cc:343] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:10.789636 6971 leader_election.cc:343] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:10.789713 7264 leader_election.cc:343] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:10.790107 7264 leader_election.cc:304] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:10.791098 7264 leader_election.cc:304] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:10.791891 8186 raft_consensus.cc:2749] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.792274 7467 tablet_bootstrap.cc:654] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:10.792455 8186 raft_consensus.cc:2804] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:10.793493 8186 raft_consensus.cc:697] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:10.794301 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:10.794401 8186 consensus_queue.cc:237] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.795095 7329 raft_consensus.cc:3060] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:10.793707 8160 leader_election.cc:290] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:10.801731 8212 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 31726fd7f40442e49b2fe0f880793206)
I20260317 21:19:10.567436 8194 raft_consensus.cc:2749] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:10.812592 8215 tablet_replica.cc:333] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:10.763638 7462 ts_tablet_manager.cc:1434] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.265s user 0.004s sys 0.002s
I20260317 21:19:10.813628 7374 heartbeater.cc:461] Registering TS with master...
I20260317 21:19:10.814072 7462 tablet_bootstrap.cc:492] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:10.815990 7329 raft_consensus.cc:2468] T 0fd3e35e25bf444cbd821ebd9f5ee018 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:10.829075 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet a4d64c48641b484ab443380808544d2f (table default.Table_aacec8c3d7ba49f9b6742baf37b423a8 [id=f966bc12b8824b708f7ff3d9f5789eb8], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a4d64c48641b484ab443380808544d2f consensus state has no leader
I20260317 21:19:10.867210 7467 tablet_bootstrap.cc:492] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:10.868044 7467 ts_tablet_manager.cc:1403] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.087s user 0.017s sys 0.023s
I20260317 21:19:10.868809 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:10.870559 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.818154 6973 leader_election.cc:304] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:10.822644 7459 tablet_bootstrap.cc:492] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:10.840619 8160 consensus_queue.cc:1048] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:10.871316 7459 ts_tablet_manager.cc:1403] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.118s user 0.013s sys 0.006s
W20260317 21:19:10.871511 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 53455.9us
I20260317 21:19:10.872085 8160 raft_consensus.cc:2804] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:10.877346 8160 raft_consensus.cc:697] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:10.878008 7459 raft_consensus.cc:359] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.878659 7459 raft_consensus.cc:385] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.878268 8160 consensus_queue.cc:237] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.878907 7459 raft_consensus.cc:740] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.879472 7459 consensus_queue.cc:260] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.884673 7467 raft_consensus.cc:359] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:10.885308 7467 raft_consensus.cc:385] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:10.885118 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_d821b351490e4553b71a09c4055e26cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:10.885556 7467 raft_consensus.cc:740] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:10.886103 7467 consensus_queue.cc:260] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:10.886647 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d821b351490e4553b71a09c4055e26cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.905566 7467 ts_tablet_manager.cc:1434] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.037s user 0.006s sys 0.000s
I20260317 21:19:10.911475 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_39092d36aadb42628dabdb2bf48c1950"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:10.913422 7467 tablet_bootstrap.cc:492] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:19:10.915558 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39092d36aadb42628dabdb2bf48c1950 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.919993 8216 tablet_replica.cc:333] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:10.921113 8216 raft_consensus.cc:2243] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:10.860881 8212 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:10.929951 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_c928476e679c41589639228020653020"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:10.939249 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c928476e679c41589639228020653020 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:10.963007 8224 raft_consensus.cc:493] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:10.964157 8224 raft_consensus.cc:515] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:10.978188 6971 consensus_peers.cc:597] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:11.001714 8215 raft_consensus.cc:2243] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:11.002759 8215 raft_consensus.cc:2272] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:11.006341 8215 ts_tablet_manager.cc:1916] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:11.011602 8216 raft_consensus.cc:2272] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:11.014233 8216 ts_tablet_manager.cc:1916] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:11.022945 7888 ts_manager.cc:194] Registered new tserver with Master: 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.057111 8212 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:11.058970 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:11.065600 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:11.072963 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:11.021869 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:11.005264 7467 tablet_bootstrap.cc:654] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.027343 8224 leader_election.cc:290] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.046799 7459 ts_tablet_manager.cc:1434] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.175s user 0.014s sys 0.021s
W20260317 21:19:11.058504 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 185551us
I20260317 21:19:11.077692 7462 tablet_bootstrap.cc:654] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:11.084395 6971 leader_election.cc:343] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.092855 8212 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 21:19:11.100476 8127 raft_consensus.cc:493] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.101876 8127 raft_consensus.cc:515] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.105183 8127 leader_election.cc:290] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.107918 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:11.109445 6971 leader_election.cc:343] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.109025 8209 tablet.cc:1723] T 9b6275af1c9c41ae8e4515deabdbdee1 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:11.098966 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:11.111014 7327 raft_consensus.cc:2468] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:11.113428 6973 leader_election.cc:304] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:11.114945 8127 raft_consensus.cc:2804] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:11.115375 8127 raft_consensus.cc:493] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:11.115818 8127 raft_consensus.cc:3060] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.118088 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_9c887813658144a5ab397c605d8954c0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:11.119184 8210 tablet.cc:1723] T 9b6275af1c9c41ae8e4515deabdbdee1 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20260317 21:19:11.119601 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c887813658144a5ab397c605d8954c0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:11.121184 7459 tablet_bootstrap.cc:492] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:11.127743 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:11.128146 7467 tablet_bootstrap.cc:492] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.128612 7467 ts_tablet_manager.cc:1403] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.216s user 0.014s sys 0.000s
I20260317 21:19:11.129118 7327 raft_consensus.cc:2468] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:11.133337 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_db13c9846b0049dca4c784a97c434893"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:11.134974 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_db13c9846b0049dca4c784a97c434893 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:11.141404 6973 leader_election.cc:304] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:11.147976 7467 raft_consensus.cc:359] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.151443 7467 raft_consensus.cc:385] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.157737 7467 raft_consensus.cc:740] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.161068 8186 raft_consensus.cc:493] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.161458 6675 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_61b756f1959b46f393c2a726704e78aa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:11.161535 8186 raft_consensus.cc:515] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.163582 8186 leader_election.cc:290] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:11.165143 7467 consensus_queue.cc:260] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.169937 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "440cb8b112044a53804357056f1226f1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:11.171988 7261 leader_election.cc:343] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.174746 7467 ts_tablet_manager.cc:1434] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.046s user 0.004s sys 0.003s
I20260317 21:19:11.175709 7467 tablet_bootstrap.cc:492] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:11.197845 8106 ts_tablet_manager.cc:1929] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20260317 21:19:11.192991 6675 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_61b756f1959b46f393c2a726704e78aa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:11.199625 8106 log.cc:1199] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:11.201062 8106 ts_tablet_manager.cc:1950] T 7eb432b6bccf4e32a2fa9d4979586c52 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:11.212527 6666 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:11.214214 6666 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:11.238091 7462 tablet_bootstrap.cc:492] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:11.227053 8224 raft_consensus.cc:2804] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:11.250553 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "440cb8b112044a53804357056f1226f1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:11.248929 8224 raft_consensus.cc:493] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:11.252982 8224 raft_consensus.cc:3060] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:11.257725 7264 leader_election.cc:343] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.258174 7264 leader_election.cc:304] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:11.259259 8186 raft_consensus.cc:2749] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:11.241413 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_9a3ec369ff814877a0ff250609d2803e [id=81dc7f934d2346cd8ad83e5d718c5d7d]) successfully deleted
I20260317 21:19:11.267457 7467 tablet_bootstrap.cc:654] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.275054 8194 raft_consensus.cc:493] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.279784 7459 tablet_bootstrap.cc:654] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.275575 8194 raft_consensus.cc:515] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.293521 8194 leader_election.cc:290] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.294983 8127 raft_consensus.cc:515] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:11.296386 7119 leader_election.cc:343] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:19:11.315367 7467 tablet_bootstrap.cc:492] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.315812 7467 ts_tablet_manager.cc:1403] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.002s
I20260317 21:19:11.318336 7467 raft_consensus.cc:359] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.318986 7467 raft_consensus.cc:385] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.319240 7467 raft_consensus.cc:740] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.298825 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd99ca9f6c540e48e29d8659fbfe777" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:11.346426 7035 raft_consensus.cc:2468] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:11.311278 8160 raft_consensus.cc:493] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.347414 8160 raft_consensus.cc:515] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.348340 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:11.349408 8160 leader_election.cc:290] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.349938 7120 leader_election.cc:304] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
W20260317 21:19:11.350484 6971 leader_election.cc:343] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.319835 7467 consensus_queue.cc:260] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.352046 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40549347f8463797d6791136883371" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:11.352862 8008 raft_consensus.cc:2804] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:11.353240 8008 raft_consensus.cc:493] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:11.353601 8008 raft_consensus.cc:3060] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:11.353513 6971 leader_election.cc:343] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:11.356299 6973 leader_election.cc:343] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3d40549347f8463797d6791136883371
I20260317 21:19:11.356818 6973 leader_election.cc:304] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:11.357568 8160 raft_consensus.cc:2749] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:11.238616 7462 ts_tablet_manager.cc:1403] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.425s user 0.011s sys 0.000s
I20260317 21:19:11.361183 7462 raft_consensus.cc:359] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.362128 7462 raft_consensus.cc:385] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.375859 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 79dc6cb9627d490498950b28fbcbba8f (DEFAULT_TABLE table=default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.376915 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79dc6cb9627d490498950b28fbcbba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.376848 6673 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]). Expected version 1 got 0
I20260317 21:19:11.331475 8224 raft_consensus.cc:515] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.381587 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:11.382205 8008 raft_consensus.cc:515] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:11.382795 6971 leader_election.cc:343] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.377760 6673 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67]). Expected version 1 got 0
I20260317 21:19:11.384263 6673 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 039d8bed256a43a6b98075b8430cbf3b (table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611]). Expected version 1 got 0
I20260317 21:19:11.385053 6673 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]). Expected version 2 got 0
I20260317 21:19:11.385608 7467 ts_tablet_manager.cc:1434] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.070s user 0.008s sys 0.000s
I20260317 21:19:11.385640 6673 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:11.392181 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 79dc6cb9627d490498950b28fbcbba8f (DEFAULT_TABLE table=default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.394212 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79dc6cb9627d490498950b28fbcbba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.395730 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 10942327694142fa82ada834b10befab (DEFAULT_TABLE table=default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.397317 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10942327694142fa82ada834b10befab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.399644 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 10942327694142fa82ada834b10befab (DEFAULT_TABLE table=default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.400795 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10942327694142fa82ada834b10befab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.400722 8238 tablet.cc:1723] T 9b6275af1c9c41ae8e4515deabdbdee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:11.404186 6670 catalog_manager.cc:5671] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:11.405822 6670 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:11.406726 6670 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:11.407514 6668 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 14c394f4ea244bee8ef04f6d519ca930
I20260317 21:19:11.410089 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 14c394f4ea244bee8ef04f6d519ca930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:10 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:11.410759 6668 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 374) took 1165 ms. Trace:
I20260317 21:19:11.413166 8127 leader_election.cc:290] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.415975 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 14c394f4ea244bee8ef04f6d519ca930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:10 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:11.426589 7467 tablet_bootstrap.cc:492] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:11.430914 6667 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 32be4818a67a41c998f04d7cedf7356e
I20260317 21:19:11.430995 6671 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Sending DeleteTablet for 3 replicas of tablet 646a115e133b43eebc1674a862ab7838
W20260317 21:19:11.431677 7119 leader_election.cc:343] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:19:11.432201 8008 leader_election.cc:290] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.432444 6671 catalog_manager.cc:6741] ReplaceTablet: tablet 646a115e133b43eebc1674a862ab7838 deleted and replaced by tablet 47ce1e486be64aa2803430a33c780348
I20260317 21:19:11.434727 6671 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39038 (request call id 368) took 1621 ms. Trace:
I20260317 21:19:11.435660 6671 rpcz_store.cc:276] 0317 21:19:09.813318 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.008572 (+195254us) service_pool.cc:225] Handling call
0317 21:19:10.069269 (+ 60697us) spinlock_profiling.cc:243] Waited 52.1 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b3bd1 00007fc05bef3227 00007fc05bef31b2 00007fc05bef30e1 00007fc05bef305c 00007fc05bef2312 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:11.434663 (+1365394us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:11.377503 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:11.377865 (+ 362us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:11.377887 (+ 22us) write_op.cc:435] Acquired schema lock
0317 21:19:11.377898 (+ 11us) tablet.cc:661] Decoding operations
0317 21:19:11.378173 (+ 275us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:11.378206 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:11.378229 (+ 23us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:11.378534 (+ 305us) tablet.cc:700] Row locks acquired
0317 21:19:11.378548 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:11.378657 (+ 109us) write_op.cc:270] Start()
0317 21:19:11.378738 (+ 81us) write_op.cc:276] Timestamp: P: 1773782351378605 usec, L: 0
0317 21:19:11.378754 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:11.379227 (+ 473us) log.cc:844] Serialized 625 byte log entry
0317 21:19:11.426842 (+ 47615us) op_driver.cc:464] REPLICATION: finished
0317 21:19:11.429037 (+ 2195us) write_op.cc:301] APPLY: starting
0317 21:19:11.429094 (+ 57us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:11.429197 (+ 103us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:11.429275 (+ 78us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:11.429630 (+ 355us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:11.429819 (+ 189us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:11.429861 (+ 42us) write_op.cc:312] APPLY: finished
0317 21:19:11.430030 (+ 169us) log.cc:844] Serialized 29 byte log entry
0317 21:19:11.430155 (+ 125us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.430226 (+ 71us) write_op.cc:454] Released schema lock
0317 21:19:11.430254 (+ 28us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":119804160,"child_traces":[["op",{"apply.queue_time_us":2072,"apply.run_cpu_time_us":1403,"apply.run_wall_time_us":1403,"num_ops":2,"prepare.queue_time_us":1947,"prepare.run_cpu_time_us":1829,"prepare.run_wall_time_us":3052,"replication_time_us":48021,"threads_started":1}]]}
I20260317 21:19:11.439792 7158 tablet_service.cc:1558] Processing DeleteTablet for tablet 32be4818a67a41c998f04d7cedf7356e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:10 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:11.440208 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 646a115e133b43eebc1674a862ab7838 with delete_type TABLET_DATA_DELETED (replaced by tablet 47ce1e486be64aa2803430a33c780348) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:11.442168 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 646a115e133b43eebc1674a862ab7838 with delete_type TABLET_DATA_DELETED (replaced by tablet 47ce1e486be64aa2803430a33c780348) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:11.442952 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 32be4818a67a41c998f04d7cedf7356e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:10 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:11.441246 6670 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:11.444402 7459 tablet_bootstrap.cc:492] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:11.444875 7459 ts_tablet_manager.cc:1403] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.324s user 0.015s sys 0.001s
I20260317 21:19:11.411137 6668 rpcz_store.cc:276] 0317 21:19:10.245734 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.277248 (+ 31514us) service_pool.cc:225] Handling call
0317 21:19:10.278511 (+ 1263us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:10.279070 (+ 559us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:10.279196 (+ 126us) catalog_manager.cc:2824] Locking tablets
0317 21:19:11.365423 (+1086227us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:11.407170 (+ 41747us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:19:11.407353 (+ 183us) catalog_manager.cc:2872] Committing in-memory state
0317 21:19:11.407437 (+ 84us) catalog_manager.cc:2876] Aborting table tasks
0317 21:19:11.410696 (+ 3259us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:11.370115 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:11.370486 (+ 371us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:11.370513 (+ 27us) write_op.cc:435] Acquired schema lock
0317 21:19:11.370525 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:11.370826 (+ 301us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:11.370877 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:11.370898 (+ 21us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:11.371150 (+ 252us) tablet.cc:700] Row locks acquired
0317 21:19:11.371163 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:11.371274 (+ 111us) write_op.cc:270] Start()
0317 21:19:11.371350 (+ 76us) write_op.cc:276] Timestamp: P: 1773782351371225 usec, L: 0
0317 21:19:11.371365 (+ 15us) op_driver.cc:348] REPLICATION: starting
0317 21:19:11.371864 (+ 499us) log.cc:844] Serialized 728 byte log entry
0317 21:19:11.387241 (+ 15377us) op_driver.cc:464] REPLICATION: finished
0317 21:19:11.387911 (+ 670us) write_op.cc:301] APPLY: starting
0317 21:19:11.387976 (+ 65us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:11.388090 (+ 114us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:11.388180 (+ 90us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:11.388427 (+ 247us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:11.388572 (+ 145us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:11.392695 (+ 4123us) write_op.cc:312] APPLY: finished
0317 21:19:11.392905 (+ 210us) log.cc:844] Serialized 29 byte log entry
0317 21:19:11.393054 (+ 149us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.393122 (+ 68us) write_op.cc:454] Released schema lock
0317 21:19:11.393153 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:11.393381 (+ 228us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.393405 (+ 24us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":517,"apply.run_cpu_time_us":1418,"apply.run_wall_time_us":5433,"num_ops":2,"prepare.queue_time_us":2221,"prepare.run_cpu_time_us":2060,"prepare.run_wall_time_us":3618,"raft.queue_time_us":92,"raft.run_cpu_time_us":566,"raft.run_wall_time_us":566,"replication_time_us":15779,"spinlock_wait_cycles":201472}]]}
I20260317 21:19:11.445592 6667 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 367) took 1736 ms. Trace:
I20260317 21:19:11.447558 6667 rpcz_store.cc:276] 0317 21:19:09.708981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:09.997058 (+288077us) service_pool.cc:225] Handling call
0317 21:19:10.068036 (+ 70978us) spinlock_profiling.cc:243] Waited 64 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1386 00007fc05bed7024 00007fc05bed6fb2 00007fc05bed6ee1 00007fc05bed6e5c 00007fc05bed6112 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:10.095680 (+ 27644us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:10.102484 (+ 6804us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:10.102676 (+ 192us) catalog_manager.cc:2824] Locking tablets
0317 21:19:11.368924 (+1266248us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:11.430487 (+ 61563us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:19:11.430700 (+ 213us) catalog_manager.cc:2872] Committing in-memory state
0317 21:19:11.430816 (+ 116us) catalog_manager.cc:2876] Aborting table tasks
0317 21:19:11.445533 (+ 14717us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:11.374065 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:11.374399 (+ 334us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:11.374426 (+ 27us) write_op.cc:435] Acquired schema lock
0317 21:19:11.374439 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:11.375214 (+ 775us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:11.375252 (+ 38us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:11.375273 (+ 21us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:11.375527 (+ 254us) tablet.cc:700] Row locks acquired
0317 21:19:11.375542 (+ 15us) write_op.cc:260] PREPARE: finished
0317 21:19:11.375650 (+ 108us) write_op.cc:270] Start()
0317 21:19:11.375728 (+ 78us) write_op.cc:276] Timestamp: P: 1773782351375602 usec, L: 0
0317 21:19:11.375744 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:11.376179 (+ 435us) log.cc:844] Serialized 759 byte log entry
0317 21:19:11.426618 (+ 50439us) op_driver.cc:464] REPLICATION: finished
0317 21:19:11.427242 (+ 624us) write_op.cc:301] APPLY: starting
0317 21:19:11.427296 (+ 54us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:11.427398 (+ 102us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:11.427528 (+ 130us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:11.427779 (+ 251us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:11.427915 (+ 136us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:11.427955 (+ 40us) write_op.cc:312] APPLY: finished
0317 21:19:11.428141 (+ 186us) log.cc:844] Serialized 29 byte log entry
0317 21:19:11.428284 (+ 143us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.428355 (+ 71us) write_op.cc:454] Released schema lock
0317 21:19:11.428390 (+ 35us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:11.428624 (+ 234us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.428722 (+ 98us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":147311488,"child_traces":[["op",{"apply.queue_time_us":474,"apply.run_cpu_time_us":1350,"apply.run_wall_time_us":1348,"num_ops":2,"prepare.queue_time_us":3286,"prepare.run_cpu_time_us":1841,"prepare.run_wall_time_us":3334,"replication_time_us":50783}]]}
I20260317 21:19:11.450603 8209 tablet.cc:1723] T fd7cf19121b5496a849118de4a29db5b P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260317 21:19:11.457583 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 99dc0fd2382142c88a80fc7eea173efc (DEFAULT_TABLE table=default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.446326 6670 catalog_manager.cc:5671] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:11.458520 7467 tablet_bootstrap.cc:654] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.459000 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99dc0fd2382142c88a80fc7eea173efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.459100 6670 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 86ae75c4ce214954beecfab9275d0ea9 (table default.Table_b6c7691408304d70b7e7aee5724bcd58 [id=242dcdafc2b74e1ca17839f96e45af88]). Expected version 1 got 0
I20260317 21:19:11.466594 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bcd99ca9f6c540e48e29d8659fbfe777" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:11.467773 7036 raft_consensus.cc:3060] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.477082 8224 leader_election.cc:290] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.479182 7459 raft_consensus.cc:359] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.479871 7459 raft_consensus.cc:385] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.480136 7459 raft_consensus.cc:740] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.482244 6667 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_7fee5e45b4aa4a93900162f41e3a3d60"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:11.483870 6667 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fee5e45b4aa4a93900162f41e3a3d60 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:11.490602 6668 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_acfbab364a9442afbd75bcbdfb2b785c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:11.495551 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 99dc0fd2382142c88a80fc7eea173efc (DEFAULT_TABLE table=default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.496755 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99dc0fd2382142c88a80fc7eea173efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.508198 8242 tablet.cc:1723] T fd7cf19121b5496a849118de4a29db5b P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20260317 21:19:11.511629 6668 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acfbab364a9442afbd75bcbdfb2b785c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:11.514096 7459 consensus_queue.cc:260] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.516343 7459 ts_tablet_manager.cc:1434] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.071s user 0.004s sys 0.000s
I20260317 21:19:11.392750 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 79dc6cb9627d490498950b28fbcbba8f (DEFAULT_TABLE table=default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.517643 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 79dc6cb9627d490498950b28fbcbba8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.411669 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 14c394f4ea244bee8ef04f6d519ca930 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:10 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:11.414577 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 10942327694142fa82ada834b10befab (DEFAULT_TABLE table=default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.521214 7467 tablet_bootstrap.cc:492] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.521010 8210 tablet.cc:1723] T fd7cf19121b5496a849118de4a29db5b P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20260317 21:19:11.521636 7467 ts_tablet_manager.cc:1403] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.095s user 0.000s sys 0.012s
I20260317 21:19:11.521714 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10942327694142fa82ada834b10befab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.414765 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:11.525628 7327 raft_consensus.cc:3060] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.436687 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet 646a115e133b43eebc1674a862ab7838 with delete_type TABLET_DATA_DELETED (replaced by tablet 47ce1e486be64aa2803430a33c780348) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:11.529565 7467 raft_consensus.cc:359] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.530309 7467 raft_consensus.cc:385] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.530571 7467 raft_consensus.cc:740] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
W20260317 21:19:11.530711 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 14c394f4ea244bee8ef04f6d519ca930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14c394f4ea244bee8ef04f6d519ca930 already in progress: creating tablet
W20260317 21:19:11.531594 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 646a115e133b43eebc1674a862ab7838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 646a115e133b43eebc1674a862ab7838 already in progress: creating tablet
I20260317 21:19:11.446343 7303 tablet_service.cc:1558] Processing DeleteTablet for tablet 32be4818a67a41c998f04d7cedf7356e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:10 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:11.481724 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:11.533162 7329 raft_consensus.cc:3060] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.537266 8008 raft_consensus.cc:493] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.537710 8008 raft_consensus.cc:515] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.539530 8008 leader_election.cc:290] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:11.540962 7120 leader_election.cc:343] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:11.544441 6669 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:11.544955 7467 consensus_queue.cc:260] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:11.546303 6669 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:45181 (request call id 69) took 1318 ms (client timeout 1000 ms). Trace:
W20260317 21:19:11.546545 6669 rpcz_store.cc:269] 0317 21:19:10.227498 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.227672 (+ 174us) service_pool.cc:225] Handling call
0317 21:19:11.546234 (+1318562us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20260317 21:19:11.547479 6669 master_service.cc:954] ReplaceTablet: received request to replace tablet 8a44a297079347bab6f8be6abe332bfa from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:11.547667 7467 ts_tablet_manager.cc:1434] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.026s user 0.003s sys 0.002s
W20260317 21:19:11.549600 6661 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 32be4818a67a41c998f04d7cedf7356e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32be4818a67a41c998f04d7cedf7356e already in progress: creating tablet
I20260317 21:19:11.551164 7327 raft_consensus.cc:2468] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:11.552938 7119 leader_election.cc:343] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:11.553493 7119 leader_election.cc:304] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:11.554667 8194 raft_consensus.cc:2749] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:11.554591 6973 leader_election.cc:304] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:11.555383 8160 raft_consensus.cc:2804] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:11.555861 8160 raft_consensus.cc:697] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:11.556747 7462 raft_consensus.cc:740] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.556699 8160 consensus_queue.cc:237] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.557183 6666 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:11.557673 7462 consensus_queue.cc:260] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.558249 6666 catalog_manager.cc:5763] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:11.560140 7462 ts_tablet_manager.cc:1434] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.201s user 0.005s sys 0.000s
I20260317 21:19:11.561523 7467 tablet_bootstrap.cc:492] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:19:11.563128 7261 consensus_peers.cc:597] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:11.563769 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 99dc0fd2382142c88a80fc7eea173efc (DEFAULT_TABLE table=default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.565071 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99dc0fd2382142c88a80fc7eea173efc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.568535 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]). Expected version 1 got 0
I20260317 21:19:11.569504 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:11.570344 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 039d8bed256a43a6b98075b8430cbf3b (table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611]). Expected version 1 got 0
I20260317 21:19:11.570931 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]). Expected version 2 got 0
I20260317 21:19:11.571523 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:11.571707 7467 tablet_bootstrap.cc:654] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.572381 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 86ae75c4ce214954beecfab9275d0ea9 (table default.Table_b6c7691408304d70b7e7aee5724bcd58 [id=242dcdafc2b74e1ca17839f96e45af88]). Expected version 1 got 0
I20260317 21:19:11.573715 6675 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67]). Expected version 1 got 0
W20260317 21:19:11.577442 6674 catalog_manager.cc:5931] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6deece45839849088dad3d873cc925b2 has no consensus state
I20260317 21:19:11.578665 7329 raft_consensus.cc:2468] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:11.581849 8194 raft_consensus.cc:493] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.582321 8194 raft_consensus.cc:515] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.584228 8194 leader_election.cc:290] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.585427 6973 leader_election.cc:304] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:11.587386 8234 raft_consensus.cc:2804] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:11.588027 8234 raft_consensus.cc:697] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:11.589432 8234 consensus_queue.cc:237] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:11.592340 7119 leader_election.cc:343] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:11.598891 7459 tablet_bootstrap.cc:492] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:11.610708 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e2f891e712341868d94bab1dd72a9e1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:11.611409 7035 raft_consensus.cc:2468] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:11.613641 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 4c19bac7772a498f87f48e6095310347 (DEFAULT_TABLE table=default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.615097 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c19bac7772a498f87f48e6095310347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.621728 7036 raft_consensus.cc:2468] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:11.624346 7467 tablet_bootstrap.cc:492] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.624099 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 4c19bac7772a498f87f48e6095310347 (DEFAULT_TABLE table=default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.624912 7467 ts_tablet_manager.cc:1403] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.008s
I20260317 21:19:11.627413 7467 raft_consensus.cc:359] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.630784 7467 raft_consensus.cc:385] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.631156 7467 raft_consensus.cc:740] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.637107 7467 consensus_queue.cc:260] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.650071 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 4c19bac7772a498f87f48e6095310347 (DEFAULT_TABLE table=default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.651239 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c19bac7772a498f87f48e6095310347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.652177 7467 ts_tablet_manager.cc:1434] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.027s user 0.003s sys 0.002s
I20260317 21:19:11.657356 7467 tablet_bootstrap.cc:492] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:11.663554 7467 tablet_bootstrap.cc:654] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.665462 7120 leader_election.cc:304] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:11.666759 8194 raft_consensus.cc:2804] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:11.667128 8194 raft_consensus.cc:493] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:11.667479 8194 raft_consensus.cc:3060] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.679461 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c19bac7772a498f87f48e6095310347. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.684029 7462 tablet_bootstrap.cc:492] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:11.684758 7120 leader_election.cc:304] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:11.686224 8008 raft_consensus.cc:2804] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:11.686693 8008 raft_consensus.cc:697] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:11.687382 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 323717acdee841aaa1795a3ca851b40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:11.687502 8008 consensus_queue.cc:237] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.689107 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 with delete_type TABLET_DATA_DELETED (replaced by tablet 6deece45839849088dad3d873cc925b2) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:11.703677 7158 tablet_service.cc:1511] Processing CreateTablet for tablet b68293996a25478b81d5202cb61e394a (DEFAULT_TABLE table=default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.704857 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68293996a25478b81d5202cb61e394a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.704844 6674 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39038 (request call id 370) took 1495 ms. Trace:
I20260317 21:19:11.707068 7305 tablet_service.cc:1511] Processing CreateTablet for tablet b68293996a25478b81d5202cb61e394a (DEFAULT_TABLE table=default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.707388 8194 raft_consensus.cc:515] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.710083 6672 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:39038 (request call id 375) took 1325 ms. Trace:
I20260317 21:19:11.710476 6672 rpcz_store.cc:276] 0317 21:19:10.384878 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.385037 (+ 159us) service_pool.cc:225] Handling call
0317 21:19:10.385954 (+ 917us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:10.389501 (+ 3547us) catalog_manager.cc:3725] Apply alter schema
0317 21:19:10.390057 (+ 556us) catalog_manager.cc:3757] Acquired catalog manager lock
0317 21:19:10.390231 (+ 174us) catalog_manager.cc:3915] Updating metadata on disk
0317 21:19:10.427701 (+ 37470us) catalog_manager.cc:3946] Committing alterations to in-memory state
0317 21:19:11.710019 (+1282318us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:10.391410 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:10.391814 (+ 404us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:10.391841 (+ 27us) write_op.cc:435] Acquired schema lock
0317 21:19:10.391855 (+ 14us) tablet.cc:661] Decoding operations
0317 21:19:10.392116 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:10.392150 (+ 34us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:10.392172 (+ 22us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:10.392315 (+ 143us) tablet.cc:700] Row locks acquired
0317 21:19:10.392327 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:10.392435 (+ 108us) write_op.cc:270] Start()
0317 21:19:10.392510 (+ 75us) write_op.cc:276] Timestamp: P: 1773782350392382 usec, L: 0
0317 21:19:10.392524 (+ 14us) op_driver.cc:348] REPLICATION: starting
0317 21:19:10.396447 (+ 3923us) log.cc:844] Serialized 405 byte log entry
0317 21:19:10.426203 (+ 29756us) op_driver.cc:464] REPLICATION: finished
0317 21:19:10.426512 (+ 309us) write_op.cc:301] APPLY: starting
0317 21:19:10.426577 (+ 65us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:10.426670 (+ 93us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:10.426749 (+ 79us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:10.426906 (+ 157us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:10.427026 (+ 120us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:10.427066 (+ 40us) write_op.cc:312] APPLY: finished
0317 21:19:10.427255 (+ 189us) log.cc:844] Serialized 23 byte log entry
0317 21:19:10.427400 (+ 145us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:10.427470 (+ 70us) write_op.cc:454] Released schema lock
0317 21:19:10.427501 (+ 31us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:10.427746 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:10.427768 (+ 22us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":160,"apply.run_cpu_time_us":1202,"apply.run_wall_time_us":1201,"num_ops":1,"prepare.queue_time_us":92,"prepare.run_cpu_time_us":1814,"prepare.run_wall_time_us":5233,"replication_time_us":33579}]]}
I20260317 21:19:11.717224 8216 ts_tablet_manager.cc:1929] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:11.717913 8216 log.cc:1199] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:11.718115 7467 tablet_bootstrap.cc:492] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.718511 7467 ts_tablet_manager.cc:1403] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.061s user 0.002s sys 0.008s
I20260317 21:19:11.718825 8216 ts_tablet_manager.cc:1950] T 7eb432b6bccf4e32a2fa9d4979586c52 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:11.721007 7467 raft_consensus.cc:359] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.721668 7467 raft_consensus.cc:385] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.721928 7467 raft_consensus.cc:740] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.722539 7467 consensus_queue.cc:260] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.735306 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68293996a25478b81d5202cb61e394a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.743532 8186 raft_consensus.cc:493] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.744004 8186 raft_consensus.cc:515] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.745811 8186 leader_election.cc:290] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:11.747272 7010 tablet_service.cc:1511] Processing CreateTablet for tablet b68293996a25478b81d5202cb61e394a (DEFAULT_TABLE table=default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:11.748080 8223 raft_consensus.cc:493] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.748328 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b68293996a25478b81d5202cb61e394a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:11.748541 8223 raft_consensus.cc:515] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.750635 8223 leader_election.cc:290] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.753875 7459 tablet_bootstrap.cc:654] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.770180 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:11.770913 7180 raft_consensus.cc:2468] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:11.776051 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:11.776680 7180 raft_consensus.cc:2468] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:11.778815 8194 leader_election.cc:290] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:11.779098 6971 consensus_peers.cc:597] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:11.785125 6971 leader_election.cc:304] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:11.796198 7261 leader_election.cc:304] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:11.797173 8186 raft_consensus.cc:2804] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:11.797523 8186 raft_consensus.cc:493] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:11.797854 8186 raft_consensus.cc:3060] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.803275 7467 ts_tablet_manager.cc:1434] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.084s user 0.010s sys 0.007s
I20260317 21:19:11.804401 7467 tablet_bootstrap.cc:492] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:11.807320 8223 raft_consensus.cc:2804] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:11.807654 8210 tablet.cc:1723] T 86ae75c4ce214954beecfab9275d0ea9 P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:19:11.808008 8223 raft_consensus.cc:493] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:11.808418 8223 raft_consensus.cc:3060] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.814569 8186 raft_consensus.cc:515] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.823393 7467 tablet_bootstrap.cc:654] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:11.833499 7119 leader_election.cc:343] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:11.835500 8160 raft_consensus.cc:493] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.827759 8259 raft_consensus.cc:493] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.839566 8259 raft_consensus.cc:515] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.841063 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3e2f891e712341868d94bab1dd72a9e1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:11.841747 7036 raft_consensus.cc:3060] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.841869 8259 leader_election.cc:290] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:11.842073 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:11.842939 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:11.843943 8256 raft_consensus.cc:493] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:19:11.844321 7261 leader_election.cc:343] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.844529 8256 raft_consensus.cc:515] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.847759 8256 leader_election.cc:290] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.836135 8160 raft_consensus.cc:515] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.853389 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8567ae10464694a1128ba2eb1d8ee1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:11.856455 7329 raft_consensus.cc:2393] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:11.860688 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:11.861512 7180 raft_consensus.cc:3060] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.866556 7467 tablet_bootstrap.cc:492] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.867033 7467 ts_tablet_manager.cc:1403] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.063s user 0.001s sys 0.009s
I20260317 21:19:11.869724 7180 raft_consensus.cc:2468] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
W20260317 21:19:11.876899 6971 leader_election.cc:343] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:11.877637 8186 leader_election.cc:290] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:11.882695 7036 raft_consensus.cc:2468] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:11.852815 8238 tablet.cc:1723] T 86ae75c4ce214954beecfab9275d0ea9 P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:19:11.869789 7467 raft_consensus.cc:359] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.890694 7467 raft_consensus.cc:385] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.891017 7467 raft_consensus.cc:740] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.891371 7261 leader_election.cc:304] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:11.891636 7467 consensus_queue.cc:260] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.892503 8186 raft_consensus.cc:2804] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:11.892786 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:11.893041 8186 raft_consensus.cc:697] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:11.893945 7467 ts_tablet_manager.cc:1434] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260317 21:19:11.894973 7467 tablet_bootstrap.cc:492] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:11.895610 7462 tablet_bootstrap.cc:654] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.887741 8255 raft_consensus.cc:493] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:11.896960 8255 raft_consensus.cc:515] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.899137 8255 leader_election.cc:290] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.902122 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:11.896513 8186 consensus_queue.cc:237] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.903847 7467 tablet_bootstrap.cc:654] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:11.907819 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:11.905686 8160 leader_election.cc:290] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:11.930795 7467 tablet_bootstrap.cc:492] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:11.931231 7467 ts_tablet_manager.cc:1403] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.037s user 0.002s sys 0.007s
I20260317 21:19:11.933688 7467 raft_consensus.cc:359] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.934614 7467 raft_consensus.cc:385] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.934957 7467 raft_consensus.cc:740] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.935649 7467 consensus_queue.cc:260] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.937424 6633 master-stress-test.cc:577] Killing test thread 1/10
I20260317 21:19:11.944380 7120 leader_election.cc:304] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:11.945168 8255 raft_consensus.cc:2804] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:11.945638 8255 raft_consensus.cc:697] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:11.946398 8255 consensus_queue.cc:237] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.948673 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:11.949481 7462 tablet_bootstrap.cc:492] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:11.949932 7462 ts_tablet_manager.cc:1403] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.290s user 0.013s sys 0.001s
I20260317 21:19:11.963315 7467 ts_tablet_manager.cc:1434] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.032s user 0.006s sys 0.001s
W20260317 21:19:11.963392 7119 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:11.964357 7467 tablet_bootstrap.cc:492] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:11.976894 7180 raft_consensus.cc:3060] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:11.983964 8215 ts_tablet_manager.cc:1929] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:11.984951 8215 log.cc:1199] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:11.986199 8215 ts_tablet_manager.cc:1950] T 7eb432b6bccf4e32a2fa9d4979586c52 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:11.987182 8215 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:47442 (request call id 286) took 1243 ms. Trace:
I20260317 21:19:11.987582 8215 rpcz_store.cc:276] 0317 21:19:10.744254 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.744976 (+ 722us) service_pool.cc:225] Handling call
0317 21:19:10.812412 (+ 67436us) ts_tablet_manager.cc:1151] Deleting tablet 7eb432b6bccf4e32a2fa9d4979586c52
0317 21:19:10.812460 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:11.516292 (+703832us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:11.983496 (+467204us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:11.987117 (+ 3621us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6441,"tablet-delete.queue_time_us":66723,"thread_start_us":70906,"threads_started":1}
I20260317 21:19:11.952267 7462 raft_consensus.cc:359] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.989028 7462 raft_consensus.cc:385] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:11.989308 7462 raft_consensus.cc:740] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:11.989985 7462 consensus_queue.cc:260] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:11.992147 7462 ts_tablet_manager.cc:1434] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.042s user 0.004s sys 0.000s
I20260317 21:19:12.011478 7467 tablet_bootstrap.cc:654] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:12.014981 7462 tablet_bootstrap.cc:492] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:12.022115 8268 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 31726fd7f40442e49b2fe0f880793206)
I20260317 21:19:12.025740 8216 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 286) took 1281 ms. Trace:
I20260317 21:19:12.026252 8216 rpcz_store.cc:276] 0317 21:19:10.744814 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.746053 (+ 1239us) service_pool.cc:225] Handling call
0317 21:19:10.919799 (+173746us) ts_tablet_manager.cc:1151] Deleting tablet 7eb432b6bccf4e32a2fa9d4979586c52
0317 21:19:10.919853 (+ 54us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:11.484977 (+565124us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:11.716875 (+231898us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:12.025666 (+308791us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":173039,"thread_start_us":140889,"threads_started":1}
I20260317 21:19:12.029899 8268 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.009299 8223 raft_consensus.cc:515] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.010673 8253 raft_consensus.cc:493] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.022809 7459 tablet_bootstrap.cc:492] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:12.063927 8223 leader_election.cc:290] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.064908 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:12.065690 7181 raft_consensus.cc:2393] T a37e53e326914575830e30915ad69cc7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:12.064837 8253 raft_consensus.cc:515] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.070932 8253 leader_election.cc:290] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.064612 7459 ts_tablet_manager.cc:1403] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.466s user 0.016s sys 0.002s
I20260317 21:19:12.071797 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:12.073299 6971 leader_election.cc:343] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.073546 7459 raft_consensus.cc:359] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.074136 7459 raft_consensus.cc:385] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.074445 7459 raft_consensus.cc:740] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.076399 7459 consensus_queue.cc:260] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.082307 7036 raft_consensus.cc:2393] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2f539d1a4a584b8ca83928be49369bf4 in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
W20260317 21:19:12.037755 6973 leader_election.cc:343] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:19:12.097460 6973 leader_election.cc:304] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:12.103451 8268 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:12.106417 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 1179e8f731604aabacaab3090a5f574c (DEFAULT_TABLE table=default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.108120 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1179e8f731604aabacaab3090a5f574c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.104316 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]) successfully deleted
W20260317 21:19:12.118346 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 14c394f4ea244bee8ef04f6d519ca930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14c394f4ea244bee8ef04f6d519ca930 already in progress: creating tablet
I20260317 21:19:12.118005 6672 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_7d9bd016aa3340c38dc1ecba85639bee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:12.101115 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet ae1b2a549af6485ca3faee3f0ad845a2 (table default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet ae1b2a549af6485ca3faee3f0ad845a2 consensus state has no leader
I20260317 21:19:12.113808 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]) successfully deleted
W20260317 21:19:12.121556 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 14c394f4ea244bee8ef04f6d519ca930 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14c394f4ea244bee8ef04f6d519ca930 already in progress: creating tablet
I20260317 21:19:12.122469 7462 tablet_bootstrap.cc:654] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:12.122365 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 646a115e133b43eebc1674a862ab7838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 646a115e133b43eebc1674a862ab7838 already in progress: creating tablet
W20260317 21:19:12.123162 6662 catalog_manager.cc:4982] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 32be4818a67a41c998f04d7cedf7356e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32be4818a67a41c998f04d7cedf7356e already in progress: creating tablet
W20260317 21:19:12.124666 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 323717acdee841aaa1795a3ca851b40f because the tablet was not found. No further retry: Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
W20260317 21:19:12.127292 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 7eb432b6bccf4e32a2fa9d4979586c52 because the tablet was not found. No further retry: Not found: Tablet not found: 7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:12.127916 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 1179e8f731604aabacaab3090a5f574c (DEFAULT_TABLE table=default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.127949 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:12.129019 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1179e8f731604aabacaab3090a5f574c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.129519 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:57 UTC) from {username='slave'} at 127.0.0.1:47442
I20260317 21:19:12.129997 7016 tablet_service.cc:1558] Processing DeleteTablet for tablet 323717acdee841aaa1795a3ca851b40f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:08 UTC) from {username='slave'} at 127.0.0.1:47442
W20260317 21:19:12.120491 6672 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9bd016aa3340c38dc1ecba85639bee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:19:12.116165 6670 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 79) took 2230 ms (client timeout 1000 ms). Trace:
I20260317 21:19:12.145319 8215 tablet_replica.cc:333] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:12.146317 8215 raft_consensus.cc:2243] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
W20260317 21:19:12.150820 6673 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 61) took 1945 ms (client timeout 1000 ms). Trace:
W20260317 21:19:12.151167 6673 rpcz_store.cc:269] 0317 21:19:10.205550 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.205696 (+ 146us) service_pool.cc:225] Handling call
0317 21:19:12.150603 (+1944907us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":39437,"reported_tablets":10}
I20260317 21:19:11.705164 6674 rpcz_store.cc:276] 0317 21:19:10.209440 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:10.209840 (+ 400us) service_pool.cc:225] Handling call
0317 21:19:10.210783 (+ 943us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:10.211475 (+ 692us) catalog_manager.cc:3725] Apply alter schema
0317 21:19:10.212030 (+ 555us) catalog_manager.cc:3757] Acquired catalog manager lock
0317 21:19:10.212218 (+ 188us) catalog_manager.cc:3915] Updating metadata on disk
0317 21:19:10.308825 (+ 96607us) catalog_manager.cc:3946] Committing alterations to in-memory state
0317 21:19:11.704777 (+1395952us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:10.216294 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:10.216704 (+ 410us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:10.216723 (+ 19us) write_op.cc:435] Acquired schema lock
0317 21:19:10.216733 (+ 10us) tablet.cc:661] Decoding operations
0317 21:19:10.216949 (+ 216us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:10.216976 (+ 27us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:10.216994 (+ 18us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:10.217111 (+ 117us) tablet.cc:700] Row locks acquired
0317 21:19:10.217123 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:10.217223 (+ 100us) write_op.cc:270] Start()
0317 21:19:10.217305 (+ 82us) write_op.cc:276] Timestamp: P: 1773782350217188 usec, L: 0
0317 21:19:10.217321 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:10.217810 (+ 489us) log.cc:844] Serialized 405 byte log entry
0317 21:19:10.303725 (+ 85915us) op_driver.cc:464] REPLICATION: finished
0317 21:19:10.304281 (+ 556us) write_op.cc:301] APPLY: starting
0317 21:19:10.304347 (+ 66us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:10.304449 (+ 102us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:10.304587 (+ 138us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:10.304810 (+ 223us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:10.304961 (+ 151us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:10.305010 (+ 49us) write_op.cc:312] APPLY: finished
0317 21:19:10.305200 (+ 190us) log.cc:844] Serialized 23 byte log entry
0317 21:19:10.305363 (+ 163us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:10.305425 (+ 62us) write_op.cc:454] Released schema lock
0317 21:19:10.305461 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:10.305697 (+ 236us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:10.305724 (+ 27us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":397,"apply.run_cpu_time_us":1374,"apply.run_wall_time_us":1374,"num_ops":1,"prepare.queue_time_us":631,"prepare.run_cpu_time_us":1711,"prepare.run_wall_time_us":1712,"replication_time_us":86303}]]}
I20260317 21:19:12.153713 6674 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e65ec8f2488d4ff99eb80d4f4f45f610" } modify_external_catalogs: true
I20260317 21:19:12.154439 6674 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_e65ec8f2488d4ff99eb80d4f4f45f610" } modify_external_catalogs: true
I20260317 21:19:12.157919 7309 tablet_service.cc:1558] Processing DeleteTablet for tablet c9b7f7c864004b9abde0a0201e196165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:12.158603 8106 tablet_replica.cc:333] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:12.159327 8106 raft_consensus.cc:2243] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:12.159989 8106 pending_rounds.cc:70] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Trying to abort 1 pending ops.
I20260317 21:19:12.160205 8106 pending_rounds.cc:77] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412508863397888 op_type: NO_OP noop_request { }
I20260317 21:19:12.160604 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 3 candidate_status { last_received { term: 2 index: 419 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:12.160851 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 57e721ac4b24438b9b2a0e2569027236 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:12.166178 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:12.160508 8106 raft_consensus.cc:2889] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:12.167147 8106 raft_consensus.cc:2272] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:12.169164 8106 ts_tablet_manager.cc:1916] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:12.180866 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b (DEFAULT_TABLE table=default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.182065 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.184271 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 86ae75c4ce214954beecfab9275d0ea9 (table default.Table_15fe620085e9499295d76e3a7b3df064 [id=242dcdafc2b74e1ca17839f96e45af88]). Expected version 2 got 0
I20260317 21:19:12.185568 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:12.186555 7007 tablet_service.cc:1511] Processing CreateTablet for tablet b1d9d0828d4646b9952b0162a97d61a8 (DEFAULT_TABLE table=default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.186834 7013 tablet_service.cc:1511] Processing CreateTablet for tablet b516c62d56fb43f2a7c73be237cac63b (DEFAULT_TABLE table=default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.187611 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d9d0828d4646b9952b0162a97d61a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.186834 7010 tablet_service.cc:1511] Processing CreateTablet for tablet 445b91a6d1294c44aea0935b34be16c0 (DEFAULT_TABLE table=default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.188622 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445b91a6d1294c44aea0935b34be16c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.190799 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b516c62d56fb43f2a7c73be237cac63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.201779 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b (DEFAULT_TABLE table=default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.202785 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.209352 7155 tablet_service.cc:1511] Processing CreateTablet for tablet 445b91a6d1294c44aea0935b34be16c0 (DEFAULT_TABLE table=default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.210357 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445b91a6d1294c44aea0935b34be16c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.220889 7459 ts_tablet_manager.cc:1434] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.149s user 0.004s sys 0.002s
I20260317 21:19:12.221388 7152 tablet_service.cc:1511] Processing CreateTablet for tablet b516c62d56fb43f2a7c73be237cac63b (DEFAULT_TABLE table=default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.221822 7459 tablet_bootstrap.cc:492] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:12.222496 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b516c62d56fb43f2a7c73be237cac63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.222450 8215 raft_consensus.cc:2272] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:12.224275 8215 ts_tablet_manager.cc:1916] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:12.225345 7158 tablet_service.cc:1511] Processing CreateTablet for tablet b1d9d0828d4646b9952b0162a97d61a8 (DEFAULT_TABLE table=default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.226339 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d9d0828d4646b9952b0162a97d61a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.224196 8256 raft_consensus.cc:493] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.236219 8256 raft_consensus.cc:515] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.236732 8276 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 31726fd7f40442e49b2fe0f880793206)
I20260317 21:19:12.237217 8276 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.245327 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 1179e8f731604aabacaab3090a5f574c (DEFAULT_TABLE table=default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.246315 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1179e8f731604aabacaab3090a5f574c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.246209 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:12.246824 7329 raft_consensus.cc:2393] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:12.238240 8256 leader_election.cc:290] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:12.257081 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 219363us
I20260317 21:19:12.257354 8253 raft_consensus.cc:2749] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.258452 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "86ae75c4ce214954beecfab9275d0ea9" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:12.152379 6673 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:39038:
name: "default.Table_3b1322b6b52a474ba8475816df4c8ee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:12.260087 6673 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b1322b6b52a474ba8475816df4c8ee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:12.260746 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 445b91a6d1294c44aea0935b34be16c0 (DEFAULT_TABLE table=default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.261478 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b (DEFAULT_TABLE table=default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.261740 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 445b91a6d1294c44aea0935b34be16c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.262405 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:12.141427 6670 rpcz_store.cc:269] 0317 21:19:09.885522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:09.994487 (+108965us) service_pool.cc:225] Handling call
0317 21:19:10.077453 (+ 82966us) spinlock_profiling.cc:243] Waited 65.8 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:12.115957 (+2038504us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:11.462206 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:11.462620 (+ 414us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:11.462643 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:19:11.462657 (+ 14us) tablet.cc:661] Decoding operations
0317 21:19:11.463021 (+ 364us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:11.463065 (+ 44us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:11.463089 (+ 24us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:11.463376 (+ 287us) tablet.cc:700] Row locks acquired
0317 21:19:11.463389 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:11.463513 (+ 124us) write_op.cc:270] Start()
0317 21:19:11.463611 (+ 98us) write_op.cc:276] Timestamp: P: 1773782351463447 usec, L: 0
0317 21:19:11.463629 (+ 18us) op_driver.cc:348] REPLICATION: starting
0317 21:19:11.478562 (+ 14933us) log.cc:844] Serialized 907 byte log entry
0317 21:19:11.479966 (+ 1404us) spinlock_profiling.cc:243] Waited 14.4 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4157a1 00007fc05c4a7d58 00007fc05c9197b1 00007fc05c918f88 00007fc05c91dc21 00007fc05c91dbd9 00007fc05c91db69 00007fc05c91db31 00007fc05c91cd5d 00000000005ab084 00000000005aaeb9
0317 21:19:11.491263 (+ 11297us) op_driver.cc:464] REPLICATION: finished
0317 21:19:11.509250 (+ 17987us) write_op.cc:301] APPLY: starting
0317 21:19:11.509312 (+ 62us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:11.509422 (+ 110us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:11.509444 (+ 22us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:11.510274 (+ 830us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:11.510427 (+ 153us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:11.510474 (+ 47us) write_op.cc:312] APPLY: finished
0317 21:19:11.510689 (+ 215us) log.cc:844] Serialized 29 byte log entry
0317 21:19:11.535196 (+ 24507us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.542632 (+ 7436us) write_op.cc:454] Released schema lock
0317 21:19:11.542678 (+ 46us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:11.542972 (+ 294us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:11.543000 (+ 28us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":426107,"reported_tablets":8,"spinlock_wait_cycles":151385856,"child_traces":[["op",{"apply.queue_time_us":17783,"apply.run_cpu_time_us":2107,"apply.run_wall_time_us":33712,"num_ops":2,"prepare.queue_time_us":110,"prepare.run_cpu_time_us":3876,"prepare.run_wall_time_us":17964,"raft.queue_time_us":4579,"raft.run_cpu_time_us":641,"raft.run_wall_time_us":642,"replication_time_us":27546,"spinlock_wait_cycles":33169664}]]}
I20260317 21:19:12.240296 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:12.263938 6670 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d86c06b3eb2849ffb26839db4313557e" } modify_external_catalogs: true
I20260317 21:19:12.264462 6670 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:39038:
table { table_name: "default.Table_d86c06b3eb2849ffb26839db4313557e" } modify_external_catalogs: true
I20260317 21:19:12.268826 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]). Expected version 1 got 0
I20260317 21:19:12.275684 7299 tablet_service.cc:1511] Processing CreateTablet for tablet b1d9d0828d4646b9952b0162a97d61a8 (DEFAULT_TABLE table=default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.277091 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1d9d0828d4646b9952b0162a97d61a8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.291541 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "57e721ac4b24438b9b2a0e2569027236" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:12.295037 7264 leader_election.cc:343] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.295234 7467 tablet_bootstrap.cc:492] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:12.295646 7467 ts_tablet_manager.cc:1403] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.331s user 0.012s sys 0.006s
I20260317 21:19:12.295480 7264 leader_election.cc:304] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:12.295804 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a37e53e326914575830e30915ad69cc7" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:12.296623 8267 raft_consensus.cc:2749] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.300853 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:12.301424 7329 raft_consensus.cc:2468] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
W20260317 21:19:12.282406 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 32be4818a67a41c998f04d7cedf7356e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 32be4818a67a41c998f04d7cedf7356e already in progress: creating tablet
I20260317 21:19:12.303241 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67]). Expected version 1 got 0
I20260317 21:19:12.304203 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]). Expected version 1 got 0
I20260317 21:19:12.307862 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 3 candidate_status { last_received { term: 2 index: 312 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:12.309543 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:12.315750 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 3 candidate_status { last_received { term: 2 index: 419 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:12.316921 7917 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 2.
I20260317 21:19:12.318315 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:12.319481 8268 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 21:19:12.319931 8268 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 31726fd7f40442e49b2fe0f880793206)
I20260317 21:19:12.320310 8268 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:19:12.320055 8276 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:12.321358 8276 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 21:19:12.329530 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]). Expected version 4 got 0
I20260317 21:19:12.334368 7462 tablet_bootstrap.cc:492] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:12.334872 7462 ts_tablet_manager.cc:1403] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.320s user 0.018s sys 0.013s
I20260317 21:19:12.335089 7303 tablet_service.cc:1511] Processing CreateTablet for tablet b516c62d56fb43f2a7c73be237cac63b (DEFAULT_TABLE table=default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.336083 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b516c62d56fb43f2a7c73be237cac63b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.336386 6671 catalog_manager.cc:5671] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f reported cstate change: term changed from 0 to 1, leader changed from <none> to d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65). New cstate: current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } }
I20260317 21:19:12.337359 7462 raft_consensus.cc:359] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.338122 7462 raft_consensus.cc:385] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.338385 7462 raft_consensus.cc:740] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.339047 7462 consensus_queue.cc:260] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.340397 6973 leader_election.cc:304] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:12.340690 7120 leader_election.cc:304] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:12.341706 7120 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:12.342065 7120 leader_election.cc:304] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:12.343241 7120 leader_election.cc:304] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:12.347198 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 047dd2548dbb4c659694b93633b39bf4 (DEFAULT_TABLE table=default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.348309 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047dd2548dbb4c659694b93633b39bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.351387 8257 raft_consensus.cc:3060] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:12.352581 7467 raft_consensus.cc:359] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.353236 7467 raft_consensus.cc:385] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.353474 7467 raft_consensus.cc:740] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.354054 7467 consensus_queue.cc:260] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.355976 7467 ts_tablet_manager.cc:1434] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.060s user 0.003s sys 0.000s
I20260317 21:19:12.360078 8257 raft_consensus.cc:2749] T 57e721ac4b24438b9b2a0e2569027236 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.350611 8255 raft_consensus.cc:2749] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.375912 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 047dd2548dbb4c659694b93633b39bf4 (DEFAULT_TABLE table=default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.382371 7462 ts_tablet_manager.cc:1434] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.047s user 0.006s sys 0.001s
I20260317 21:19:12.383473 7462 tablet_bootstrap.cc:492] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:12.383819 8253 raft_consensus.cc:2749] T a37e53e326914575830e30915ad69cc7 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.384450 8253 consensus_queue.cc:1048] T 0fd3e35e25bf444cbd821ebd9f5ee018 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:12.386040 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 44628.2us
I20260317 21:19:12.386680 6973 leader_election.cc:304] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:12.387319 8253 raft_consensus.cc:2804] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:12.387632 8253 raft_consensus.cc:493] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:12.387960 8253 raft_consensus.cc:3060] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:12.395328 8253 raft_consensus.cc:515] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.395812 7467 tablet_bootstrap.cc:492] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:12.397611 8255 raft_consensus.cc:493] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.398097 8255 raft_consensus.cc:515] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.400378 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 047dd2548dbb4c659694b93633b39bf4 (DEFAULT_TABLE table=default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.401473 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047dd2548dbb4c659694b93633b39bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.399020 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:12.403357 6971 leader_election.cc:343] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.403395 8255 leader_election.cc:290] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.410468 7302 tablet_service.cc:1511] Processing CreateTablet for tablet a8a02da14e4c4e0a9ce93e60255c68a9 (DEFAULT_TABLE table=default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.411187 7009 tablet_service.cc:1511] Processing CreateTablet for tablet a8a02da14e4c4e0a9ce93e60255c68a9 (DEFAULT_TABLE table=default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.411568 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a02da14e4c4e0a9ce93e60255c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:12.412993 7119 leader_election.cc:343] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:19:12.382125 6667 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39038 (request call id 389) took 1032 ms. Trace:
I20260317 21:19:12.416853 6667 rpcz_store.cc:276] 0317 21:19:11.349398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.451106 (+101708us) service_pool.cc:225] Handling call
0317 21:19:11.482131 (+ 31025us) spinlock_profiling.cc:243] Waited 26.6 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1076 00007fc05bed2524 00007fc05bed24b2 00007fc05bed23e1 00007fc05bed235c 00007fc05bed1612 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:11.484271 (+ 2140us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:11.487334 (+ 3063us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:12.381227 (+893893us) catalog_manager.cc:2207] Wrote table and tablets to system table
0317 21:19:12.381595 (+ 368us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0317 21:19:12.381934 (+ 339us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:11.759282 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:11.759717 (+ 435us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:11.759743 (+ 26us) write_op.cc:435] Acquired schema lock
0317 21:19:11.759756 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:11.760017 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:11.760057 (+ 40us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:11.760078 (+ 21us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:11.760253 (+ 175us) tablet.cc:700] Row locks acquired
0317 21:19:11.760265 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:11.760370 (+ 105us) write_op.cc:270] Start()
0317 21:19:11.760446 (+ 76us) write_op.cc:276] Timestamp: P: 1773782351760285 usec, L: 0
0317 21:19:11.760459 (+ 13us) op_driver.cc:348] REPLICATION: starting
0317 21:19:11.760983 (+ 524us) log.cc:844] Serialized 433 byte log entry
0317 21:19:12.327752 (+566769us) op_driver.cc:464] REPLICATION: finished
0317 21:19:12.378640 (+ 50888us) write_op.cc:301] APPLY: starting
0317 21:19:12.378733 (+ 93us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:12.378842 (+ 109us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:12.378955 (+ 113us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:12.379450 (+ 495us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:12.379617 (+ 167us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:12.379664 (+ 47us) write_op.cc:312] APPLY: finished
0317 21:19:12.379914 (+ 250us) log.cc:844] Serialized 29 byte log entry
0317 21:19:12.380719 (+ 805us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:12.380811 (+ 92us) write_op.cc:454] Released schema lock
0317 21:19:12.380852 (+ 41us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":61239680,"child_traces":[["op",{"apply.queue_time_us":50670,"apply.run_cpu_time_us":1878,"apply.run_wall_time_us":2486,"num_ops":2,"prepare.queue_time_us":270542,"prepare.run_cpu_time_us":1998,"prepare.run_wall_time_us":2739,"raft.queue_time_us":84,"raft.run_cpu_time_us":593,"raft.run_wall_time_us":593,"replication_time_us":567140,"thread_start_us":72725,"threads_started":1}]]}
I20260317 21:19:12.422890 7462 tablet_bootstrap.cc:654] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:12.412990 8256 raft_consensus.cc:2749] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.430192 6633 master-stress-test.cc:579] Killed test thread 1/10
I20260317 21:19:12.430472 6633 master-stress-test.cc:577] Killing test thread 2/10
I20260317 21:19:12.437912 8253 leader_election.cc:290] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.443699 7467 tablet_bootstrap.cc:654] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:12.443661 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:12.444180 7328 raft_consensus.cc:3060] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:12.444972 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 101782us
W20260317 21:19:12.446372 6659 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 646a115e133b43eebc1674a862ab7838 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 646a115e133b43eebc1674a862ab7838 already in progress: creating tablet
I20260317 21:19:12.452117 7328 raft_consensus.cc:2468] T 2fd1d8d70e9849fa9e107b2dd3355c58 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:12.453363 8277 tablet.cc:1723] T 86ae75c4ce214954beecfab9275d0ea9 P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20260317 21:19:12.459969 7298 tablet_service.cc:1511] Processing CreateTablet for tablet fe51fd1fdd864b6f8b44cb45ba332c4d (DEFAULT_TABLE table=default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.441780 8234 consensus_queue.cc:1048] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:12.449101 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 047dd2548dbb4c659694b93633b39bf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.461452 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe51fd1fdd864b6f8b44cb45ba332c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.462363 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a02da14e4c4e0a9ce93e60255c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.462767 8281 raft_consensus.cc:493] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.463302 8281 raft_consensus.cc:515] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:12.468230 7119 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:12.468911 8281 leader_election.cc:290] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:12.469646 7119 consensus_peers.cc:597] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: bcd99ca9f6c540e48e29d8659fbfe777. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:12.476815 8268 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.453017 7459 tablet_bootstrap.cc:654] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:12.457589 7006 tablet_service.cc:1511] Processing CreateTablet for tablet fe51fd1fdd864b6f8b44cb45ba332c4d (DEFAULT_TABLE table=default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.478890 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe51fd1fdd864b6f8b44cb45ba332c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.479079 6759 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 FOLLOWER]: Rejecting Update request from peer 31726fd7f40442e49b2fe0f880793206 for earlier term 2. Current term is 3. Ops: [2.421-2.428]
I20260317 21:19:12.479755 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 3 candidate_status { last_received { term: 2 index: 420 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:19:12.481978 8278 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 }, Status: INVALID_TERM, Last received: 2.420, Next index: 421, Last known committed idx: 419, Time since last communication: 0.001s
I20260317 21:19:12.482730 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 3 candidate_status { last_received { term: 2 index: 420 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
I20260317 21:19:12.483165 7684 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 LEADER]: Stepping down as leader of term 2
I20260317 21:19:12.483268 7917 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:19:12.483533 7684 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 LEADER]: Becoming Follower/Learner. State: Replica: 31726fd7f40442e49b2fe0f880793206, State: Running, Role: LEADER
I20260317 21:19:12.484280 7684 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 420, Committed index: 420, Last appended: 2.428, Last appended by leader: 428, Current term: 2, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.486249 7684 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:19:12.491290 7917 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 3.
I20260317 21:19:12.493420 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:12.497438 8106 ts_tablet_manager.cc:1929] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:12.500684 6671 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]). Expected version 2 got 0
I20260317 21:19:12.502961 7154 tablet_service.cc:1511] Processing CreateTablet for tablet a8a02da14e4c4e0a9ce93e60255c68a9 (DEFAULT_TABLE table=default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.504745 8268 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 3 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:12.511423 8268 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 FOLLOWER]: Leader election won for term 3
I20260317 21:19:12.514134 8268 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 LEADER]: Becoming Leader. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:12.520737 7007 tablet_service.cc:1511] Processing CreateTablet for tablet b03b54b989d04e0990ee3ccfe7a104a4 (DEFAULT_TABLE table=default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.521870 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03b54b989d04e0990ee3ccfe7a104a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.521359 8234 raft_consensus.cc:493] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.522169 7467 tablet_bootstrap.cc:492] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:12.522586 7467 ts_tablet_manager.cc:1403] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.127s user 0.004s sys 0.006s
I20260317 21:19:12.522634 8234 raft_consensus.cc:515] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.526814 8234 leader_election.cc:290] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.526772 7152 tablet_service.cc:1511] Processing CreateTablet for tablet fe51fd1fdd864b6f8b44cb45ba332c4d (DEFAULT_TABLE table=default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.526772 7158 tablet_service.cc:1511] Processing CreateTablet for tablet f044d4c56cf24215a7f0beec13ece53a (DEFAULT_TABLE table=default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.529637 7013 tablet_service.cc:1511] Processing CreateTablet for tablet f044d4c56cf24215a7f0beec13ece53a (DEFAULT_TABLE table=default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.530769 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f044d4c56cf24215a7f0beec13ece53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.533070 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d41fc13b04ec688fc38a78f55e83d" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:12.533964 8268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 419, Committed index: 419, Last appended: 2.420, Last appended by leader: 420, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.534708 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a02da14e4c4e0a9ce93e60255c68a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.527604 7155 tablet_service.cc:1511] Processing CreateTablet for tablet b03b54b989d04e0990ee3ccfe7a104a4 (DEFAULT_TABLE table=default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:12.534984 6971 leader_election.cc:343] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.536022 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03b54b989d04e0990ee3ccfe7a104a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.529956 7305 tablet_service.cc:1511] Processing CreateTablet for tablet b03b54b989d04e0990ee3ccfe7a104a4 (DEFAULT_TABLE table=default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.537206 8215 ts_tablet_manager.cc:1929] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:12.537424 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b03b54b989d04e0990ee3ccfe7a104a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.537878 8215 log.cc:1199] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:12.538651 8215 ts_tablet_manager.cc:1950] T 3e2f891e712341868d94bab1dd72a9e1 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:12.540160 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 3e2f891e712341868d94bab1dd72a9e1 (table default.Table_0c31d9e5b0bd4454acccb920047a5e73 [id=8fbd9827525947d58d75c279e4b92b6a]) successfully deleted
I20260317 21:19:12.541791 8215 tablet_replica.cc:333] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:12.543021 8215 raft_consensus.cc:2243] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:12.543623 8215 raft_consensus.cc:2272] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:12.544854 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d41fc13b04ec688fc38a78f55e83d" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:12.545434 7327 raft_consensus.cc:2468] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:12.546994 6973 leader_election.cc:304] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:12.548663 6973 leader_election.cc:304] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:12.549068 8223 raft_consensus.cc:2804] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:12.555737 8234 raft_consensus.cc:2804] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:12.556133 8234 raft_consensus.cc:493] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:12.555892 7459 tablet_bootstrap.cc:492] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:12.556697 8234 raft_consensus.cc:3060] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:12.557703 7459 ts_tablet_manager.cc:1403] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.336s user 0.008s sys 0.004s
I20260317 21:19:12.563793 8106 log.cc:1199] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/c9b7f7c864004b9abde0a0201e196165
I20260317 21:19:12.564626 8106 ts_tablet_manager.cc:1950] T c9b7f7c864004b9abde0a0201e196165 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:12.567601 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet c9b7f7c864004b9abde0a0201e196165 (table default.Table_ce47367ed77d4c06ad8c624c31dd3b2b [id=7f6c80768baf4e05aad94f1b86cfdfdd]) successfully deleted
E20260317 21:19:12.508837 6671 catalog_manager.cc:5826] Error updating tablets from d3cb3b49e3e94537a6e434ea4e037c3f: Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "fd7cf19121b5496a849118de4a29db5b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27322 live_row_count: 0 } } updated_tablets { tablet_id: "564a02888d854ea58d7a590cba816b86" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27353 live_row_count: 0 } } updated_tablets { tablet_id: "3d40549347f8463797d6791136883371" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: UNKNOWN } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "57e721ac4b24438b9b2a0e2569027236" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c9b7f7c864004b9abde0a0201e196165" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "bcd99ca9f6c540e48e29d8659fbfe777" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "7eb432b6bccf4e32a2fa9d4979586c52" state: STOPPED consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_DELETED } updated_tablets { tablet_id: "3e2f891e712341868d94bab1dd72a9e1" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a37e53e326914575830e30915ad69cc7" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "221c71b908f942ca97f111f857f0335a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27322 live_row_count: 0 } } updated_tablets { tablet_id: "86ae75c4ce214954beecfab9275d0ea9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { } } updated_tablets { tablet_id: "63a27911e0e545f5811eb075997d0b0f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27244 live_row_count: 0 } } removed_tablet_ids: "323717acdee841aaa1795a3ca851b40f" removed_tablet_ids: "a662b365ec4846bf9e9ee243d41cb91a" sequence_number: 79
I20260317 21:19:12.569621 7462 tablet_bootstrap.cc:492] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:12.570035 7462 ts_tablet_manager.cc:1403] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.187s user 0.006s sys 0.007s
W20260317 21:19:12.570920 6671 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:49239 (request call id 80) took 1144 ms (client timeout 1000 ms). Trace:
W20260317 21:19:12.571262 6671 rpcz_store.cc:269] 0317 21:19:11.426455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.443099 (+ 16644us) service_pool.cc:225] Handling call
0317 21:19:11.493540 (+ 50441us) spinlock_profiling.cc:243] Waited 18.5 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2afb2e 00007fc05becb4a4 00007fc05becb432 00007fc05becb361 00007fc05becb2dc 00007fc05beca592 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:12.508554 (+1015014us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:12.508575 (+ 21us) write_op.cc:454] Released schema lock
0317 21:19:12.570851 (+ 62276us) inbound_call.cc:173] Queueing failure response
Related trace 'op':
0317 21:19:12.508316 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER)
0317 21:19:12.508367 (+ 51us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:12.508387 (+ 20us) write_op.cc:454] Released schema lock
0317 21:19:12.508414 (+ 27us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":452468,"reported_tablets":17,"spinlock_wait_cycles":42625408,"child_traces":[["op",{}]]}
I20260317 21:19:12.574777 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe51fd1fdd864b6f8b44cb45ba332c4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.574993 7462 raft_consensus.cc:359] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.575696 7462 raft_consensus.cc:385] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.575968 7462 raft_consensus.cc:740] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.576464 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f044d4c56cf24215a7f0beec13ece53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.577139 7462 consensus_queue.cc:260] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.562975 7467 raft_consensus.cc:359] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.582417 7467 raft_consensus.cc:385] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.582152 8234 raft_consensus.cc:515] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.582715 7467 raft_consensus.cc:740] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.583321 7467 consensus_queue.cc:260] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.576088 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]). Expected version 1 got 0
I20260317 21:19:12.591929 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]). Expected version 1 got 0
I20260317 21:19:12.594467 8285 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 3 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:12.595300 8285 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:12.604228 8215 ts_tablet_manager.cc:1916] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:12.607645 6667 catalog_manager.cc:5671] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67). New cstate: current_term: 1 leader_uuid: "2f539d1a4a584b8ca83928be49369bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } }
I20260317 21:19:12.610126 7467 ts_tablet_manager.cc:1434] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.087s user 0.007s sys 0.000s
I20260317 21:19:12.611064 7467 tablet_bootstrap.cc:492] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:12.611060 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 8d94620882364ed3929ee314814910c9 (table default.Table_38fe3ad6f1bf4796af58d99b53c46560 [id=872cbb89b0d1405e9858d37dc22c05c4]). Expected version 2 got 0
I20260317 21:19:12.629905 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]). Expected version 2 got 0
I20260317 21:19:12.630703 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 039d8bed256a43a6b98075b8430cbf3b (table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611]). Expected version 1 got 0
I20260317 21:19:12.631353 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]). Expected version 1 got 0
I20260317 21:19:12.632242 6667 catalog_manager.cc:5763] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877) does not have the latest schema for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67]). Expected version 1 got 0
I20260317 21:19:12.634830 8234 leader_election.cc:290] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.579655 7462 ts_tablet_manager.cc:1434] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260317 21:19:12.639767 8223 raft_consensus.cc:697] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:12.640780 8223 consensus_queue.cc:237] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.644024 8106 tablet_replica.cc:333] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:12.644922 8106 raft_consensus.cc:2243] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:12.644977 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d41fc13b04ec688fc38a78f55e83d" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:12.645450 8106 raft_consensus.cc:2272] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:12.647177 8106 ts_tablet_manager.cc:1916] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:12.647141 7181 raft_consensus.cc:3060] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:12.669340 7009 tablet_service.cc:1511] Processing CreateTablet for tablet b35816e1de7d4b13927848538cdc3e78 (DEFAULT_TABLE table=default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.670586 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b35816e1de7d4b13927848538cdc3e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.674955 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 585a5f0834614eb1aa4c3e405fa9bbc8 (DEFAULT_TABLE table=default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.676025 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585a5f0834614eb1aa4c3e405fa9bbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.648294 7462 tablet_bootstrap.cc:492] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:12.659336 7302 tablet_service.cc:1511] Processing CreateTablet for tablet f044d4c56cf24215a7f0beec13ece53a (DEFAULT_TABLE table=default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.681492 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f044d4c56cf24215a7f0beec13ece53a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.670594 7303 tablet_service.cc:1511] Processing CreateTablet for tablet b35816e1de7d4b13927848538cdc3e78 (DEFAULT_TABLE table=default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.685286 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b35816e1de7d4b13927848538cdc3e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.674029 7154 tablet_service.cc:1511] Processing CreateTablet for tablet b35816e1de7d4b13927848538cdc3e78 (DEFAULT_TABLE table=default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.691002 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b35816e1de7d4b13927848538cdc3e78. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.691002 7462 tablet_bootstrap.cc:654] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:12.703629 7459 raft_consensus.cc:359] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.704300 7459 raft_consensus.cc:385] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.704566 7459 raft_consensus.cc:740] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.705277 7459 consensus_queue.cc:260] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.707857 7459 ts_tablet_manager.cc:1434] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.149s user 0.006s sys 0.000s
I20260317 21:19:12.715569 7467 tablet_bootstrap.cc:654] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:12.717921 6661 consensus_peers.cc:183] SignalRequest failed: Illegal state: peer closed
I20260317 21:19:12.719154 7459 tablet_bootstrap.cc:492] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:12.720160 6971 leader_election.cc:343] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.721086 8288 consensus_queue.cc:1048] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:19:12.722770 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 174194us
I20260317 21:19:12.734025 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8567ae10464694a1128ba2eb1d8ee1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:12.734578 7036 raft_consensus.cc:2468] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:12.735656 8291 consensus_queue.cc:1048] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:12.736824 7120 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:12.737205 7120 leader_election.cc:304] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:12.737721 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "564d41fc13b04ec688fc38a78f55e83d" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:12.738250 7327 raft_consensus.cc:3060] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:12.738185 7120 leader_election.cc:304] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:12.739331 8256 raft_consensus.cc:2749] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:12.739904 8256 raft_consensus.cc:2804] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:12.723362 6997 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:47438 (request call id 75) took 1074 ms. Trace:
I20260317 21:19:12.740242 8256 raft_consensus.cc:493] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:12.740573 8256 raft_consensus.cc:3060] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:12.740531 6997 rpcz_store.cc:276] 0317 21:19:11.648851 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.649022 (+ 171us) service_pool.cc:225] Handling call
0317 21:19:11.900606 (+251584us) spinlock_profiling.cc:243] Waited 88.5 ms on lock 0x7b54000f0854. stack: 00007f15f787f423 00007f15f787f320 00007f15f73f4574 0000000000509544 00000000005094d9 00007f15fe8da7a1 00007f15fe96ae28 00007f15ff9ab79d 00007f15fe133104 00007f15fe133092 00007f15fe132fc1 00007f15fe132f3c 00007f15fe1321f2 00007f15fc66d34c 00007f15fc66c786 00007f15fc66c12f
0317 21:19:12.086509 (+185903us) spinlock_profiling.cc:243] Waited 53.4 ms on lock 0x7b5400172fd4. stack: 00007f15f787f423 00007f15f787f320 00007f15f73f4574 0000000000509544 00000000005094d9 00007f15fe8da7a1 00007f15fe96ae28 00007f15ff9ab79d 00007f15fe133104 00007f15fe133092 00007f15fe132fc1 00007f15fe132f3c 00007f15fe1321f2 00007f15fc66d34c 00007f15fc66c786 00007f15fc66c12f
0317 21:19:12.723272 (+636763us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":326541824}
I20260317 21:19:12.746881 8186 raft_consensus.cc:493] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.747362 8186 raft_consensus.cc:515] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.749413 8186 leader_election.cc:290] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:12.751343 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2dccb915fad4666bd7978f8c416db44" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:12.753047 7261 leader_election.cc:343] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.756810 7327 raft_consensus.cc:2468] T 564d41fc13b04ec688fc38a78f55e83d P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:12.765707 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 585a5f0834614eb1aa4c3e405fa9bbc8 (DEFAULT_TABLE table=default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.766899 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585a5f0834614eb1aa4c3e405fa9bbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.769378 8256 raft_consensus.cc:515] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:12.773542 7119 leader_election.cc:343] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:19:12.775347 6973 leader_election.cc:304] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:12.776136 8289 raft_consensus.cc:2804] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:12.776695 8289 raft_consensus.cc:697] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:12.777819 8289 consensus_queue.cc:237] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.783597 7459 tablet_bootstrap.cc:654] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:12.784235 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fc8567ae10464694a1128ba2eb1d8ee1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:12.784940 7038 raft_consensus.cc:3060] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
E20260317 21:19:12.635156 6667 catalog_manager.cc:5826] Error updating tablets from 2f539d1a4a584b8ca83928be49369bf4: Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER. Tablet report was: is_incremental: true updated_tablets { tablet_id: "99c6483618044272890b59f6b7447081" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "fd7cf19121b5496a849118de4a29db5b" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 4 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "564a02888d854ea58d7a590cba816b86" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "9b6275af1c9c41ae8e4515deabdbdee1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "c0ef92fef382401a9564c17e0c223f6a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 1 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "440cb8b112044a53804357056f1226f1" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "86ae75c4ce214954beecfab9275d0ea9" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 2 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "63a27911e0e545f5811eb075997d0b0f" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "c9b7f7c864004b9abde0a0201e196165" state: STOPPED consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_DELETED } updated_tablets { tablet_id: "039d8bed256a43a6b98075b8430cbf3b" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "2fd1d8d70e9849fa9e107b2dd3355c58" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "57e721ac4b24438b9b2a0e2569027236" state: RUNNING consensus_state { current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "8d94620882364ed3929ee314814910c9" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "a37e53e326914575830e30915ad69cc7" state: RUNNING consensus_state { current_term: 1 leader_uuid: "2f539d1a4a584b8ca83928be49369bf4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } health_report { overall_health: HEALTHY } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY stats { on_disk_size: 27244 live_row_count: 0 } } updated_tablets { tablet_id: "221c71b908f942ca97f111f857f0335a" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "6caf05c79a5f470e8b7796d0e85e36f1" state: RUNNING consensus_state { current_term: 1 leader_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 3 tablet_data_state: TABLET_DATA_READY } updated_tablets { tablet_id: "564d41fc13b04ec688fc38a78f55e83d" state: RUNNING consensus_state { current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } } } } schema_version: 0 tablet_data_state: TABLET_DATA_READY } removed_tablet_ids: "7eb432b6bccf4e32a2fa9d4979586c52" removed_tablet_ids: "323717acdee841aaa1795a3ca851b40f" sequence_number: 62
I20260317 21:19:12.792208 7462 tablet_bootstrap.cc:492] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:12.792888 7462 ts_tablet_manager.cc:1403] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.145s user 0.011s sys 0.000s
I20260317 21:19:12.794991 7038 raft_consensus.cc:2468] T fc8567ae10464694a1128ba2eb1d8ee1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:12.795373 7462 raft_consensus.cc:359] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.795991 7462 raft_consensus.cc:385] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.796259 7462 raft_consensus.cc:740] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.794198 6667 master_service.cc:954] ReplaceTablet: received request to replace tablet 8a44a297079347bab6f8be6abe332bfa from {username='slave'} at 127.0.0.1:39038
I20260317 21:19:12.800065 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 585a5f0834614eb1aa4c3e405fa9bbc8 (DEFAULT_TABLE table=default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:12.808902 7120 consensus_peers.cc:597] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:12.809659 7386 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.122.126:40107)
I20260317 21:19:12.809881 7120 leader_election.cc:304] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:12.811870 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2dccb915fad4666bd7978f8c416db44" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:12.826469 7298 tablet_service.cc:1511] Processing CreateTablet for tablet ec5aa338aa2a41f0a7aeb4194189208f (DEFAULT_TABLE table=default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.826753 7007 tablet_service.cc:1511] Processing CreateTablet for tablet ec5aa338aa2a41f0a7aeb4194189208f (DEFAULT_TABLE table=default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:12.828181 7264 leader_election.cc:343] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:12.828608 7264 leader_election.cc:304] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:12.829888 8186 raft_consensus.cc:2749] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:12.831955 6973 consensus_peers.cc:597] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:12.840952 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec5aa338aa2a41f0a7aeb4194189208f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.851410 8255 raft_consensus.cc:2804] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:12.852213 8255 raft_consensus.cc:697] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:12.852254 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 76bd0f5369cf400299039e4631ed11b0 (DEFAULT_TABLE table=default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.857875 7013 tablet_service.cc:1511] Processing CreateTablet for tablet d2aba7eba6df430e86e003ef3a179912 (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.858541 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 585a5f0834614eb1aa4c3e405fa9bbc8. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.848320 8215 ts_tablet_manager.cc:1929] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:12.858982 8215 log.cc:1199] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:19:12.861755 8255 consensus_queue.cc:237] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.852690 8281 raft_consensus.cc:493] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.864132 8281 raft_consensus.cc:515] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.864900 8215 ts_tablet_manager.cc:1950] T bcd99ca9f6c540e48e29d8659fbfe777 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:12.866192 8281 leader_election.cc:290] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.871225 7158 tablet_service.cc:1511] Processing CreateTablet for tablet d2aba7eba6df430e86e003ef3a179912 (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.865130 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bd0f5369cf400299039e4631ed11b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.872288 7158 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2aba7eba6df430e86e003ef3a179912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.872291 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2aba7eba6df430e86e003ef3a179912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.876766 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:12.877676 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 76bd0f5369cf400299039e4631ed11b0 (DEFAULT_TABLE table=default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.878751 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bd0f5369cf400299039e4631ed11b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.881897 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec5aa338aa2a41f0a7aeb4194189208f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.884248 7684 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.884773 7684 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.889758 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 4 candidate_status { last_received { term: 2 index: 428 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:12.891777 7684 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:12.893155 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 4 candidate_status { last_received { term: 2 index: 428 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:12.893735 7917 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31726fd7f40442e49b2fe0f880793206 in term 3.
I20260317 21:19:12.895076 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291
I20260317 21:19:12.896049 7684 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 21:19:12.896435 7684 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:12.896845 7684 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 3 FOLLOWER]: Advancing to term 4
W20260317 21:19:12.899928 6659 catalog_manager.cc:5075] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): alter failed for tablet 7eb432b6bccf4e32a2fa9d4979586c52 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]),no further retry: Not found: Tablet not found: 7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:12.772251 8256 leader_election.cc:290] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:12.909346 7155 tablet_service.cc:1511] Processing CreateTablet for tablet ec5aa338aa2a41f0a7aeb4194189208f (DEFAULT_TABLE table=default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.910404 7155 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec5aa338aa2a41f0a7aeb4194189208f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.910485 7459 tablet_bootstrap.cc:492] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:12.910902 7459 ts_tablet_manager.cc:1403] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.202s user 0.007s sys 0.003s
I20260317 21:19:12.916982 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 76bd0f5369cf400299039e4631ed11b0 (DEFAULT_TABLE table=default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.918076 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76bd0f5369cf400299039e4631ed11b0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.931489 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 4f33574134934f048632aae269244368 (DEFAULT_TABLE table=default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.932602 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f33574134934f048632aae269244368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.935556 8186 raft_consensus.cc:493] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:12.936053 8186 raft_consensus.cc:515] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.942708 7459 raft_consensus.cc:359] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.943372 7459 raft_consensus.cc:385] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:12.943630 7459 raft_consensus.cc:740] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:12.944381 7462 consensus_queue.cc:260] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.950304 7684 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:12.953542 7459 consensus_queue.cc:260] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:12.957341 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 4f33574134934f048632aae269244368 (DEFAULT_TABLE table=default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:12.960804 7459 ts_tablet_manager.cc:1434] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.050s user 0.005s sys 0.002s
I20260317 21:19:12.993862 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f33574134934f048632aae269244368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:12.997539 7459 tablet_bootstrap.cc:492] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:12.999256 7467 tablet_bootstrap.cc:492] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:12.999672 7467 ts_tablet_manager.cc:1403] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.389s user 0.010s sys 0.000s
I20260317 21:19:13.000334 7010 tablet_service.cc:1511] Processing CreateTablet for tablet b492743dfc4d48398fbae0ee56882aa3 (DEFAULT_TABLE table=default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.001420 7010 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b492743dfc4d48398fbae0ee56882aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.002046 7467 raft_consensus.cc:359] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.002794 7467 raft_consensus.cc:385] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.003046 7467 raft_consensus.cc:740] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.004197 7152 tablet_service.cc:1511] Processing CreateTablet for tablet b492743dfc4d48398fbae0ee56882aa3 (DEFAULT_TABLE table=default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.005275 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b492743dfc4d48398fbae0ee56882aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.008015 7459 tablet_bootstrap.cc:654] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.012956 7467 consensus_queue.cc:260] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.015609 7467 ts_tablet_manager.cc:1434] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260317 21:19:13.015803 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 4 candidate_status { last_received { term: 2 index: 428 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:19:13.016557 7467 tablet_bootstrap.cc:492] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:13.018713 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 4 candidate_status { last_received { term: 2 index: 428 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
W20260317 21:19:13.018915 7119 consensus_peers.cc:597] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e2f891e712341868d94bab1dd72a9e1. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:13.019237 7917 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:19:13.020443 7302 tablet_service.cc:1511] Processing CreateTablet for tablet d2aba7eba6df430e86e003ef3a179912 (DEFAULT_TABLE table=default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.021695 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d2aba7eba6df430e86e003ef3a179912. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.022873 6662 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet bcd99ca9f6c540e48e29d8659fbfe777 (table default.Table_fcec6e585d6245ff828c8c0c98450d63 [id=d238f6cb466c49d6af5ae380e967b054]) successfully deleted
W20260317 21:19:13.029443 6662 catalog_manager.cc:4969] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): delete failed for tablet 323717acdee841aaa1795a3ca851b40f because the tablet was not found. No further retry: Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:13.030596 6683 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 FOLLOWER]: Rejecting Update request from peer 21186091f2c9410ea6e585c5ef2c6291 for earlier term 3. Current term is 4. Ops: []
I20260317 21:19:13.032438 8285 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 }, Status: INVALID_TERM, Last received: 0.0, Next index: 421, Last known committed idx: 420, Time since last communication: 0.001s
I20260317 21:19:13.033394 8285 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Peer 31726fd7f40442e49b2fe0f880793206 log is divergent from this leader: its last log entry 2.428 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 420
I20260317 21:19:13.033869 8285 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 LEADER]: Stepping down as leader of term 3
I20260317 21:19:13.034143 8285 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 LEADER]: Becoming Follower/Learner. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:13.034997 8285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 419, Committed index: 419, Last appended: 3.421, Last appended by leader: 421, Current term: 3, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:13.036566 8285 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:19:13.041531 7917 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 31726fd7f40442e49b2fe0f880793206 in term 4.
I20260317 21:19:13.042906 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291
I20260317 21:19:13.048776 8186 leader_election.cc:290] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:13.048902 7462 ts_tablet_manager.cc:1434] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.256s user 0.007s sys 0.000s
I20260317 21:19:13.049333 7299 tablet_service.cc:1511] Processing CreateTablet for tablet b492743dfc4d48398fbae0ee56882aa3 (DEFAULT_TABLE table=default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.049566 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1a68e92ca8bf4cd6ae0f4453c06c35a7" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:13.050465 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b492743dfc4d48398fbae0ee56882aa3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.048900 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 4f33574134934f048632aae269244368 (DEFAULT_TABLE table=default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.053424 7119 leader_election.cc:304] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:13.054257 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f33574134934f048632aae269244368. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.054471 8255 raft_consensus.cc:2749] T 1a68e92ca8bf4cd6ae0f4453c06c35a7 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:13.055927 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d94620882364ed3929ee314814910c9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:13.057251 7264 leader_election.cc:343] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.059311 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d94620882364ed3929ee314814910c9" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:13.066550 8289 raft_consensus.cc:493] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.067044 8289 raft_consensus.cc:515] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.069829 8289 leader_election.cc:290] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.070290 7684 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 4 election: Requested vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:13.070832 7684 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 FOLLOWER]: Leader election won for term 4
I20260317 21:19:13.071213 7684 raft_consensus.cc:697] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 LEADER]: Becoming Leader. State: Replica: 31726fd7f40442e49b2fe0f880793206, State: Running, Role: LEADER
I20260317 21:19:13.071669 8106 ts_tablet_manager.cc:1929] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:13.072130 8106 log.cc:1199] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/57e721ac4b24438b9b2a0e2569027236
I20260317 21:19:13.073297 8106 ts_tablet_manager.cc:1950] T 57e721ac4b24438b9b2a0e2569027236 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
W20260317 21:19:13.074800 7120 consensus_peers.cc:597] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3e2f891e712341868d94bab1dd72a9e1. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:13.080441 7684 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 420, Committed index: 420, Last appended: 2.428, Last appended by leader: 428, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:13.086195 7013 tablet_service.cc:1511] Processing CreateTablet for tablet 3130fb84aeed42c0bba58b970150c983 (DEFAULT_TABLE table=default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.091065 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:13.094936 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 57e721ac4b24438b9b2a0e2569027236 (table default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab]) successfully deleted
W20260317 21:19:13.095242 6971 leader_election.cc:343] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.096460 7013 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3130fb84aeed42c0bba58b970150c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.091264 7467 tablet_bootstrap.cc:654] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.101704 7459 tablet_bootstrap.cc:492] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:13.102198 7459 ts_tablet_manager.cc:1403] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.105s user 0.012s sys 0.000s
I20260317 21:19:13.107928 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 3130fb84aeed42c0bba58b970150c983 (DEFAULT_TABLE table=default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.109368 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3130fb84aeed42c0bba58b970150c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.112576 8313 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31726fd7f40442e49b2fe0f880793206. Latest consensus state: current_term: 4 leader_uuid: "31726fd7f40442e49b2fe0f880793206" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:13.113401 8313 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:13.117781 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 9531ac1602894c43b9c165d9a5eddc76 (DEFAULT_TABLE table=default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.119412 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9531ac1602894c43b9c165d9a5eddc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.127033 7016 tablet_service.cc:1511] Processing CreateTablet for tablet 06f6a1c205a74045a9f79daf98809b45 (DEFAULT_TABLE table=default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.128271 7016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f6a1c205a74045a9f79daf98809b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.047609 8307 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.133530 8307 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:09.581413 7918 rpcz_store.cc:276] 0317 21:19:05.673353 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:05.674237 (+ 884us) service_pool.cc:225] Handling call
0317 21:19:06.459266 (+785029us) raft_consensus.cc:1471] Updating replica for 307 ops
0317 21:19:06.766659 (+307393us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:06.766692 (+ 33us) write_op.cc:454] Released schema lock
0317 21:19:06.781654 (+ 14962us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:06.781680 (+ 26us) write_op.cc:454] Released schema lock
0317 21:19:06.802019 (+ 20339us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:06.802048 (+ 29us) write_op.cc:454] Released schema lock
0317 21:19:06.837503 (+ 35455us) raft_consensus.cc:1534] Early marking committed up to index 5
0317 21:19:06.957816 (+120313us) raft_consensus.cc:1539] Triggering prepare for 307 ops
0317 21:19:06.959190 (+ 1374us) write_op.cc:270] Start()
0317 21:19:06.959344 (+ 154us) write_op.cc:276] Timestamp: P: 1773782328886197 usec, L: 0
0317 21:19:07.017257 (+ 57913us) write_op.cc:270] Start()
0317 21:19:07.017360 (+ 103us) write_op.cc:276] Timestamp: P: 1773782328889278 usec, L: 0
0317 21:19:07.018003 (+ 643us) write_op.cc:270] Start()
0317 21:19:07.018094 (+ 91us) write_op.cc:276] Timestamp: P: 1773782328892213 usec, L: 0
0317 21:19:07.019345 (+ 1251us) write_op.cc:270] Start()
0317 21:19:07.019430 (+ 85us) write_op.cc:276] Timestamp: P: 1773782328958130 usec, L: 0
0317 21:19:07.019942 (+ 512us) write_op.cc:270] Start()
0317 21:19:07.020023 (+ 81us) write_op.cc:276] Timestamp: P: 1773782328972459 usec, L: 0
0317 21:19:07.020527 (+ 504us) write_op.cc:270] Start()
0317 21:19:07.035008 (+ 14481us) write_op.cc:276] Timestamp: P: 1773782328992725 usec, L: 0
0317 21:19:07.035769 (+ 761us) write_op.cc:270] Start()
0317 21:19:07.035879 (+ 110us) write_op.cc:276] Timestamp: P: 1773782329036441 usec, L: 0
0317 21:19:07.037019 (+ 1140us) write_op.cc:270] Start()
0317 21:19:07.037115 (+ 96us) write_op.cc:276] Timestamp: P: 1773782329103638 usec, L: 0
0317 21:19:07.037639 (+ 524us) write_op.cc:270] Start()
0317 21:19:07.037723 (+ 84us) write_op.cc:276] Timestamp: P: 1773782329163892 usec, L: 0
0317 21:19:07.038195 (+ 472us) write_op.cc:270] Start()
0317 21:19:07.038267 (+ 72us) write_op.cc:276] Timestamp: P: 1773782329196053 usec, L: 0
0317 21:19:07.038701 (+ 434us) write_op.cc:270] Start()
0317 21:19:07.038776 (+ 75us) write_op.cc:276] Timestamp: P: 1773782329218180 usec, L: 0
0317 21:19:07.039440 (+ 664us) write_op.cc:270] Start()
0317 21:19:07.039546 (+ 106us) write_op.cc:276] Timestamp: P: 1773782329226081 usec, L: 0
0317 21:19:07.040019 (+ 473us) write_op.cc:270] Start()
0317 21:19:07.040096 (+ 77us) write_op.cc:276] Timestamp: P: 1773782329235535 usec, L: 0
0317 21:19:07.134396 (+ 94300us) write_op.cc:270] Start()
0317 21:19:07.134509 (+ 113us) write_op.cc:276] Timestamp: P: 1773782329238169 usec, L: 0
0317 21:19:07.135108 (+ 599us) write_op.cc:270] Start()
0317 21:19:07.135189 (+ 81us) write_op.cc:276] Timestamp: P: 1773782329341978 usec, L: 0
0317 21:19:07.141762 (+ 6573us) write_op.cc:270] Start()
0317 21:19:07.141880 (+ 118us) write_op.cc:276] Timestamp: P: 1773782329369647 usec, L: 0
0317 21:19:07.142478 (+ 598us) write_op.cc:270] Start()
0317 21:19:07.142578 (+ 100us) write_op.cc:276] Timestamp: P: 1773782329387223 usec, L: 0
0317 21:19:07.143172 (+ 594us) write_op.cc:270] Start()
0317 21:19:07.143257 (+ 85us) write_op.cc:276] Timestamp: P: 1773782329398050 usec, L: 0
0317 21:19:07.143770 (+ 513us) write_op.cc:270] Start()
0317 21:19:07.143851 (+ 81us) write_op.cc:276] Timestamp: P: 1773782329443531 usec, L: 0
0317 21:19:07.144383 (+ 532us) write_op.cc:270] Start()
0317 21:19:07.144460 (+ 77us) write_op.cc:276] Timestamp: P: 1773782329510490 usec, L: 0
0317 21:19:07.203847 (+ 59387us) write_op.cc:270] Start()
0317 21:19:07.204065 (+ 218us) write_op.cc:276] Timestamp: P: 1773782329517121 usec, L: 0
0317 21:19:07.204802 (+ 737us) write_op.cc:270] Start()
0317 21:19:07.204911 (+ 109us) write_op.cc:276] Timestamp: P: 1773782329546677 usec, L: 0
0317 21:19:07.205642 (+ 731us) write_op.cc:270] Start()
0317 21:19:07.205746 (+ 104us) write_op.cc:276] Timestamp: P: 1773782329617310 usec, L: 0
0317 21:19:07.208134 (+ 2388us) write_op.cc:270] Start()
0317 21:19:07.208281 (+ 147us) write_op.cc:276] Timestamp: P: 1773782329636160 usec, L: 0
0317 21:19:07.245034 (+ 36753us) write_op.cc:270] Start()
0317 21:19:07.245141 (+ 107us) write_op.cc:276] Timestamp: P: 1773782329757949 usec, L: 0
0317 21:19:07.245817 (+ 676us) write_op.cc:270] Start()
0317 21:19:07.245932 (+ 115us) write_op.cc:276] Timestamp: P: 1773782329803655 usec, L: 0
0317 21:19:07.246428 (+ 496us) write_op.cc:270] Start()
0317 21:19:07.246509 (+ 81us) write_op.cc:276] Timestamp: P: 1773782329834398 usec, L: 0
0317 21:19:07.247156 (+ 647us) write_op.cc:270] Start()
0317 21:19:07.247240 (+ 84us) write_op.cc:276] Timestamp: P: 1773782329953796 usec, L: 0
0317 21:19:07.247740 (+ 500us) write_op.cc:270] Start()
0317 21:19:07.247824 (+ 84us) write_op.cc:276] Timestamp: P: 1773782329986415 usec, L: 0
0317 21:19:07.248350 (+ 526us) write_op.cc:270] Start()
0317 21:19:07.248432 (+ 82us) write_op.cc:276] Timestamp: P: 1773782329995184 usec, L: 0
0317 21:19:07.321401 (+ 72969us) write_op.cc:270] Start()
0317 21:19:07.321543 (+ 142us) write_op.cc:276] Timestamp: P: 1773782330015315 usec, L: 0
0317 21:19:07.322278 (+ 735us) write_op.cc:270] Start()
0317 21:19:07.322394 (+ 116us) write_op.cc:276] Timestamp: P: 1773782330018320 usec, L: 0
0317 21:19:07.323088 (+ 694us) write_op.cc:270] Start()
0317 21:19:07.323197 (+ 109us) write_op.cc:276] Timestamp: P: 1773782330089057 usec, L: 0
0317 21:19:07.323899 (+ 702us) write_op.cc:270] Start()
0317 21:19:07.324014 (+ 115us) write_op.cc:276] Timestamp: P: 1773782330109527 usec, L: 0
0317 21:19:07.413093 (+ 89079us) write_op.cc:270] Start()
0317 21:19:07.413252 (+ 159us) write_op.cc:276] Timestamp: P: 1773782330210611 usec, L: 0
0317 21:19:07.414133 (+ 881us) write_op.cc:270] Start()
0317 21:19:07.414237 (+ 104us) write_op.cc:276] Timestamp: P: 1773782330320219 usec, L: 0
0317 21:19:07.414962 (+ 725us) write_op.cc:270] Start()
0317 21:19:07.415046 (+ 84us) write_op.cc:276] Timestamp: P: 1773782330327229 usec, L: 0
0317 21:19:07.415654 (+ 608us) write_op.cc:270] Start()
0317 21:19:07.415749 (+ 95us) write_op.cc:276] Timestamp: P: 1773782330331081 usec, L: 0
0317 21:19:07.416358 (+ 609us) write_op.cc:270] Start()
0317 21:19:07.416435 (+ 77us) write_op.cc:276] Timestamp: P: 1773782330365499 usec, L: 0
0317 21:19:07.416928 (+ 493us) write_op.cc:270] Start()
0317 21:19:07.417006 (+ 78us) write_op.cc:276] Timestamp: P: 1773782330408115 usec, L: 0
0317 21:19:07.417498 (+ 492us) write_op.cc:270] Start()
0317 21:19:07.417579 (+ 81us) write_op.cc:276] Timestamp: P: 1773782330420845 usec, L: 0
0317 21:19:07.418041 (+ 462us) write_op.cc:270] Start()
0317 21:19:07.418119 (+ 78us) write_op.cc:276] Timestamp: P: 1773782330442083 usec, L: 0
0317 21:19:07.418555 (+ 436us) write_op.cc:270] Start()
0317 21:19:07.418643 (+ 88us) write_op.cc:276] Timestamp: P: 1773782330481479 usec, L: 0
0317 21:19:07.419232 (+ 589us) write_op.cc:270] Start()
0317 21:19:07.419312 (+ 80us) write_op.cc:276] Timestamp: P: 1773782330540359 usec, L: 0
0317 21:19:07.420278 (+ 966us) write_op.cc:270] Start()
0317 21:19:07.420376 (+ 98us) write_op.cc:276] Timestamp: P: 1773782330585845 usec, L: 0
0317 21:19:07.420963 (+ 587us) write_op.cc:270] Start()
0317 21:19:07.421035 (+ 72us) write_op.cc:276] Timestamp: P: 1773782330608804 usec, L: 0
0317 21:19:07.421548 (+ 513us) write_op.cc:270] Start()
0317 21:19:07.421625 (+ 77us) write_op.cc:276] Timestamp: P: 1773782330632999 usec, L: 0
0317 21:19:07.422103 (+ 478us) write_op.cc:270] Start()
0317 21:19:07.422253 (+ 150us) write_op.cc:276] Timestamp: P: 1773782330693536 usec, L: 0
0317 21:19:07.422765 (+ 512us) write_op.cc:270] Start()
0317 21:19:07.422846 (+ 81us) write_op.cc:276] Timestamp: P: 1773782330713298 usec, L: 0
0317 21:19:07.423295 (+ 449us) write_op.cc:270] Start()
0317 21:19:07.423372 (+ 77us) write_op.cc:276] Timestamp: P: 1773782330904174 usec, L: 0
0317 21:19:07.423810 (+ 438us) write_op.cc:270] Start()
0317 21:19:07.423899 (+ 89us) write_op.cc:276] Timestamp: P: 1773782330908191 usec, L: 0
0317 21:19:07.424483 (+ 584us) write_op.cc:270] Start()
0317 21:19:07.424582 (+ 99us) write_op.cc:276] Timestamp: P: 1773782330916196 usec, L: 0
0317 21:19:07.425173 (+ 591us) write_op.cc:270] Start()
0317 21:19:07.425273 (+ 100us) write_op.cc:276] Timestamp: P: 1773782330928110 usec, L: 0
0317 21:19:07.425962 (+ 689us) write_op.cc:270] Start()
0317 21:19:07.426050 (+ 88us) write_op.cc:276] Timestamp: P: 1773782331007252 usec, L: 0
0317 21:19:07.426560 (+ 510us) write_op.cc:270] Start()
0317 21:19:07.426644 (+ 84us) write_op.cc:276] Timestamp: P: 1773782331011405 usec, L: 0
0317 21:19:07.427169 (+ 525us) write_op.cc:270] Start()
0317 21:19:07.427258 (+ 89us) write_op.cc:276] Timestamp: P: 1773782331264544 usec, L: 0
0317 21:19:07.427821 (+ 563us) write_op.cc:270] Start()
0317 21:19:07.427925 (+ 104us) write_op.cc:276] Timestamp: P: 1773782331297818 usec, L: 0
0317 21:19:07.428423 (+ 498us) write_op.cc:270] Start()
0317 21:19:07.428508 (+ 85us) write_op.cc:276] Timestamp: P: 1773782331357823 usec, L: 0
0317 21:19:07.429177 (+ 669us) write_op.cc:270] Start()
0317 21:19:07.429284 (+ 107us) write_op.cc:276] Timestamp: P: 1773782331415710 usec, L: 0
0317 21:19:07.430357 (+ 1073us) write_op.cc:270] Start()
0317 21:19:07.430448 (+ 91us) write_op.cc:276] Timestamp: P: 1773782331473575 usec, L: 0
0317 21:19:07.431022 (+ 574us) write_op.cc:270] Start()
0317 21:19:07.431101 (+ 79us) write_op.cc:276] Timestamp: P: 1773782331500925 usec, L: 0
0317 21:19:07.431616 (+ 515us) write_op.cc:270] Start()
0317 21:19:07.431700 (+ 84us) write_op.cc:276] Timestamp: P: 1773782331531809 usec, L: 0
0317 21:19:07.432175 (+ 475us) write_op.cc:270] Start()
0317 21:19:07.432258 (+ 83us) write_op.cc:276] Timestamp: P: 1773782331578708 usec, L: 0
0317 21:19:07.432806 (+ 548us) write_op.cc:270] Start()
0317 21:19:07.432895 (+ 89us) write_op.cc:276] Timestamp: P: 1773782331581360 usec, L: 0
0317 21:19:07.433517 (+ 622us) write_op.cc:270] Start()
0317 21:19:07.433606 (+ 89us) write_op.cc:276] Timestamp: P: 1773782331583713 usec, L: 0
0317 21:19:07.434152 (+ 546us) write_op.cc:270] Start()
0317 21:19:07.434236 (+ 84us) write_op.cc:276] Timestamp: P: 1773782331612237 usec, L: 0
0317 21:19:07.434730 (+ 494us) write_op.cc:270] Start()
0317 21:19:07.434819 (+ 89us) write_op.cc:276] Timestamp: P: 1773782331652177 usec, L: 0
0317 21:19:07.435326 (+ 507us) write_op.cc:270] Start()
0317 21:19:07.435411 (+ 85us) write_op.cc:276] Timestamp: P: 1773782331792969 usec, L: 0
0317 21:19:07.435972 (+ 561us) write_op.cc:270] Start()
0317 21:19:07.436076 (+ 104us) write_op.cc:276] Timestamp: P: 1773782331796457 usec, L: 0
0317 21:19:07.436816 (+ 740us) write_op.cc:270] Start()
0317 21:19:07.436912 (+ 96us) write_op.cc:276] Timestamp: P: 1773782331831448 usec, L: 0
0317 21:19:07.437445 (+ 533us) write_op.cc:270] Start()
0317 21:19:07.437539 (+ 94us) write_op.cc:276] Timestamp: P: 1773782331851850 usec, L: 0
0317 21:19:07.438074 (+ 535us) write_op.cc:270] Start()
0317 21:19:07.438168 (+ 94us) write_op.cc:276] Timestamp: P: 1773782331959908 usec, L: 0
0317 21:19:07.438754 (+ 586us) write_op.cc:270] Start()
0317 21:19:07.438852 (+ 98us) write_op.cc:276] Timestamp: P: 1773782331978550 usec, L: 0
0317 21:19:07.439370 (+ 518us) write_op.cc:270] Start()
0317 21:19:07.439458 (+ 88us) write_op.cc:276] Timestamp: P: 1773782331982300 usec, L: 0
0317 21:19:07.440100 (+ 642us) write_op.cc:270] Start()
0317 21:19:07.440197 (+ 97us) write_op.cc:276] Timestamp: P: 1773782331986316 usec, L: 0
0317 21:19:07.440840 (+ 643us) write_op.cc:270] Start()
0317 21:19:07.440930 (+ 90us) write_op.cc:276] Timestamp: P: 1773782332107664 usec, L: 0
0317 21:19:07.441425 (+ 495us) write_op.cc:270] Start()
0317 21:19:07.441511 (+ 86us) write_op.cc:276] Timestamp: P: 1773782332198198 usec, L: 0
0317 21:19:07.442027 (+ 516us) write_op.cc:270] Start()
0317 21:19:07.442207 (+ 180us) write_op.cc:276] Timestamp: P: 1773782332201523 usec, L: 0
0317 21:19:07.442700 (+ 493us) write_op.cc:270] Start()
0317 21:19:07.442801 (+ 101us) write_op.cc:276] Timestamp: P: 1773782332250708 usec, L: 0
0317 21:19:07.443300 (+ 499us) write_op.cc:270] Start()
0317 21:19:07.443391 (+ 91us) write_op.cc:276] Timestamp: P: 1773782332282577 usec, L: 0
0317 21:19:07.454859 (+ 11468us) write_op.cc:270] Start()
0317 21:19:07.454954 (+ 95us) write_op.cc:276] Timestamp: P: 1773782332299719 usec, L: 0
0317 21:19:07.455604 (+ 650us) write_op.cc:270] Start()
0317 21:19:07.456121 (+ 517us) write_op.cc:276] Timestamp: P: 1773782332417135 usec, L: 0
0317 21:19:07.456781 (+ 660us) write_op.cc:270] Start()
0317 21:19:07.456869 (+ 88us) write_op.cc:276] Timestamp: P: 1773782332451614 usec, L: 0
0317 21:19:07.457495 (+ 626us) write_op.cc:270] Start()
0317 21:19:07.457584 (+ 89us) write_op.cc:276] Timestamp: P: 1773782332468070 usec, L: 0
0317 21:19:07.458163 (+ 579us) write_op.cc:270] Start()
0317 21:19:07.458253 (+ 90us) write_op.cc:276] Timestamp: P: 1773782332575245 usec, L: 0
0317 21:19:07.458775 (+ 522us) write_op.cc:270] Start()
0317 21:19:07.458869 (+ 94us) write_op.cc:276] Timestamp: P: 1773782332601827 usec, L: 0
0317 21:19:07.459357 (+ 488us) write_op.cc:270] Start()
0317 21:19:07.459445 (+ 88us) write_op.cc:276] Timestamp: P: 1773782332644918 usec, L: 0
0317 21:19:07.459934 (+ 489us) write_op.cc:270] Start()
0317 21:19:07.460021 (+ 87us) write_op.cc:276] Timestamp: P: 1773782332690829 usec, L: 0
0317 21:19:07.460494 (+ 473us) write_op.cc:270] Start()
0317 21:19:07.460580 (+ 86us) write_op.cc:276] Timestamp: P: 1773782332742133 usec, L: 0
0317 21:19:07.461108 (+ 528us) write_op.cc:270] Start()
0317 21:19:07.461199 (+ 91us) write_op.cc:276] Timestamp: P: 1773782332761790 usec, L: 0
0317 21:19:07.461746 (+ 547us) write_op.cc:270] Start()
0317 21:19:07.461841 (+ 95us) write_op.cc:276] Timestamp: P: 1773782332908327 usec, L: 0
0317 21:19:07.462367 (+ 526us) write_op.cc:270] Start()
0317 21:19:07.462560 (+ 193us) write_op.cc:276] Timestamp: P: 1773782333033081 usec, L: 0
0317 21:19:07.463058 (+ 498us) write_op.cc:270] Start()
0317 21:19:07.463144 (+ 86us) write_op.cc:276] Timestamp: P: 1773782333036938 usec, L: 0
0317 21:19:07.464715 (+ 1571us) write_op.cc:270] Start()
0317 21:19:07.464825 (+ 110us) write_op.cc:276] Timestamp: P: 1773782333104557 usec, L: 0
0317 21:19:07.465326 (+ 501us) write_op.cc:270] Start()
0317 21:19:07.465412 (+ 86us) write_op.cc:276] Timestamp: P: 1773782333108273 usec, L: 0
0317 21:19:07.466866 (+ 1454us) write_op.cc:270] Start()
0317 21:19:07.466966 (+ 100us) write_op.cc:276] Timestamp: P: 1773782333218680 usec, L: 0
0317 21:19:07.467553 (+ 587us) write_op.cc:270] Start()
0317 21:19:07.467645 (+ 92us) write_op.cc:276] Timestamp: P: 1773782333360423 usec, L: 0
0317 21:19:07.468364 (+ 719us) write_op.cc:270] Start()
0317 21:19:07.468595 (+ 231us) write_op.cc:276] Timestamp: P: 1773782333453764 usec, L: 0
0317 21:19:07.469361 (+ 766us) write_op.cc:270] Start()
0317 21:19:07.469457 (+ 96us) write_op.cc:276] Timestamp: P: 1773782333464666 usec, L: 0
0317 21:19:07.470009 (+ 552us) write_op.cc:270] Start()
0317 21:19:07.470099 (+ 90us) write_op.cc:276] Timestamp: P: 1773782333490912 usec, L: 0
0317 21:19:07.470622 (+ 523us) write_op.cc:270] Start()
0317 21:19:07.470719 (+ 97us) write_op.cc:276] Timestamp: P: 1773782333565950 usec, L: 0
0317 21:19:07.471429 (+ 710us) write_op.cc:270] Start()
0317 21:19:07.471549 (+ 120us) write_op.cc:276] Timestamp: P: 1773782333589341 usec, L: 0
0317 21:19:07.472223 (+ 674us) write_op.cc:270] Start()
0317 21:19:07.472314 (+ 91us) write_op.cc:276] Timestamp: P: 1773782333593262 usec, L: 0
0317 21:19:07.474296 (+ 1982us) write_op.cc:270] Start()
0317 21:19:07.474392 (+ 96us) write_op.cc:276] Timestamp: P: 1773782333822337 usec, L: 0
0317 21:19:07.474912 (+ 520us) write_op.cc:270] Start()
0317 21:19:07.475001 (+ 89us) write_op.cc:276] Timestamp: P: 1773782333826698 usec, L: 0
0317 21:19:07.475493 (+ 492us) write_op.cc:270] Start()
0317 21:19:07.475571 (+ 78us) write_op.cc:276] Timestamp: P: 1773782333836626 usec, L: 0
0317 21:19:07.476060 (+ 489us) write_op.cc:270] Start()
0317 21:19:07.476139 (+ 79us) write_op.cc:276] Timestamp: P: 1773782333865350 usec, L: 0
0317 21:19:07.476612 (+ 473us) write_op.cc:270] Start()
0317 21:19:07.476722 (+ 110us) write_op.cc:276] Timestamp: P: 1773782334001440 usec, L: 0
0317 21:19:07.477413 (+ 691us) write_op.cc:270] Start()
0317 21:19:07.477505 (+ 92us) write_op.cc:276] Timestamp: P: 1773782334054295 usec, L: 0
0317 21:19:07.478163 (+ 658us) write_op.cc:270] Start()
0317 21:19:07.478254 (+ 91us) write_op.cc:276] Timestamp: P: 1773782334063467 usec, L: 0
0317 21:19:07.478747 (+ 493us) write_op.cc:270] Start()
0317 21:19:07.478831 (+ 84us) write_op.cc:276] Timestamp: P: 1773782334154980 usec, L: 0
0317 21:19:07.479324 (+ 493us) write_op.cc:270] Start()
0317 21:19:07.479412 (+ 88us) write_op.cc:276] Timestamp: P: 1773782334189367 usec, L: 0
0317 21:19:07.480136 (+ 724us) write_op.cc:270] Start()
0317 21:19:07.480220 (+ 84us) write_op.cc:276] Timestamp: P: 1773782334195791 usec, L: 0
0317 21:19:07.480863 (+ 643us) write_op.cc:270] Start()
0317 21:19:07.480932 (+ 69us) write_op.cc:276] Timestamp: P: 1773782334199994 usec, L: 0
0317 21:19:07.481332 (+ 400us) write_op.cc:270] Start()
0317 21:19:07.481398 (+ 66us) write_op.cc:276] Timestamp: P: 1773782334261793 usec, L: 0
0317 21:19:07.482081 (+ 683us) write_op.cc:270] Start()
0317 21:19:07.482151 (+ 70us) write_op.cc:276] Timestamp: P: 1773782334314746 usec, L: 0
0317 21:19:07.482548 (+ 397us) write_op.cc:270] Start()
0317 21:19:07.482614 (+ 66us) write_op.cc:276] Timestamp: P: 1773782334317476 usec, L: 0
0317 21:19:07.483841 (+ 1227us) write_op.cc:270] Start()
0317 21:19:07.483907 (+ 66us) write_op.cc:276] Timestamp: P: 1773782334476416 usec, L: 0
0317 21:19:07.484301 (+ 394us) write_op.cc:270] Start()
0317 21:19:07.484364 (+ 63us) write_op.cc:276] Timestamp: P: 1773782334522650 usec, L: 0
0317 21:19:07.484827 (+ 463us) write_op.cc:270] Start()
0317 21:19:07.484893 (+ 66us) write_op.cc:276] Timestamp: P: 1773782334570159 usec, L: 0
0317 21:19:07.485309 (+ 416us) write_op.cc:270] Start()
0317 21:19:07.485384 (+ 75us) write_op.cc:276] Timestamp: P: 1773782334628595 usec, L: 0
0317 21:19:07.486664 (+ 1280us) write_op.cc:270] Start()
0317 21:19:07.486735 (+ 71us) write_op.cc:276] Timestamp: P: 1773782334682488 usec, L: 0
0317 21:19:07.487174 (+ 439us) write_op.cc:270] Start()
0317 21:19:07.487246 (+ 72us) write_op.cc:276] Timestamp: P: 1773782334685790 usec, L: 0
0317 21:19:07.487684 (+ 438us) write_op.cc:270] Start()
0317 21:19:07.487764 (+ 80us) write_op.cc:276] Timestamp: P: 1773782334769073 usec, L: 0
0317 21:19:07.488242 (+ 478us) write_op.cc:270] Start()
0317 21:19:07.488316 (+ 74us) write_op.cc:276] Timestamp: P: 1773782334772841 usec, L: 0
0317 21:19:07.488806 (+ 490us) write_op.cc:270] Start()
0317 21:19:07.488881 (+ 75us) write_op.cc:276] Timestamp: P: 1773782334802738 usec, L: 0
0317 21:19:07.489317 (+ 436us) write_op.cc:270] Start()
0317 21:19:07.489383 (+ 66us) write_op.cc:276] Timestamp: P: 1773782334909871 usec, L: 0
0317 21:19:07.489859 (+ 476us) write_op.cc:270] Start()
0317 21:19:07.489925 (+ 66us) write_op.cc:276] Timestamp: P: 1773782334972792 usec, L: 0
0317 21:19:07.490483 (+ 558us) write_op.cc:270] Start()
0317 21:19:07.490550 (+ 67us) write_op.cc:276] Timestamp: P: 1773782335069851 usec, L: 0
0317 21:19:07.491226 (+ 676us) write_op.cc:270] Start()
0317 21:19:07.491294 (+ 68us) write_op.cc:276] Timestamp: P: 1773782335073510 usec, L: 0
0317 21:19:07.491854 (+ 560us) write_op.cc:270] Start()
0317 21:19:07.491932 (+ 78us) write_op.cc:276] Timestamp: P: 1773782335079305 usec, L: 0
0317 21:19:07.492343 (+ 411us) write_op.cc:270] Start()
0317 21:19:07.492418 (+ 75us) write_op.cc:276] Timestamp: P: 1773782335085964 usec, L: 0
0317 21:19:07.492903 (+ 485us) write_op.cc:270] Start()
0317 21:19:07.492971 (+ 68us) write_op.cc:276] Timestamp: P: 1773782335303975 usec, L: 0
0317 21:19:07.493406 (+ 435us) write_op.cc:270] Start()
0317 21:19:07.493498 (+ 92us) write_op.cc:276] Timestamp: P: 1773782335306190 usec, L: 0
0317 21:19:07.494098 (+ 600us) write_op.cc:270] Start()
0317 21:19:07.494304 (+ 206us) write_op.cc:276] Timestamp: P: 1773782335326689 usec, L: 0
0317 21:19:07.494770 (+ 466us) write_op.cc:270] Start()
0317 21:19:07.494837 (+ 67us) write_op.cc:276] Timestamp: P: 1773782335369870 usec, L: 0
0317 21:19:07.495280 (+ 443us) write_op.cc:270] Start()
0317 21:19:07.495353 (+ 73us) write_op.cc:276] Timestamp: P: 1773782335465252 usec, L: 0
0317 21:19:07.495793 (+ 440us) write_op.cc:270] Start()
0317 21:19:07.495859 (+ 66us) write_op.cc:276] Timestamp: P: 1773782335629362 usec, L: 0
0317 21:19:07.527593 (+ 31734us) write_op.cc:270] Start()
0317 21:19:07.527695 (+ 102us) write_op.cc:276] Timestamp: P: 1773782335647180 usec, L: 0
0317 21:19:07.528205 (+ 510us) write_op.cc:270] Start()
0317 21:19:07.528281 (+ 76us) write_op.cc:276] Timestamp: P: 1773782335737169 usec, L: 0
0317 21:19:07.528758 (+ 477us) write_op.cc:270] Start()
0317 21:19:07.528834 (+ 76us) write_op.cc:276] Timestamp: P: 1773782335751622 usec, L: 0
0317 21:19:07.529285 (+ 451us) write_op.cc:270] Start()
0317 21:19:07.529359 (+ 74us) write_op.cc:276] Timestamp: P: 1773782335756158 usec, L: 0
0317 21:19:07.529783 (+ 424us) write_op.cc:270] Start()
0317 21:19:07.529856 (+ 73us) write_op.cc:276] Timestamp: P: 1773782335852894 usec, L: 0
0317 21:19:07.530316 (+ 460us) write_op.cc:270] Start()
0317 21:19:07.530398 (+ 82us) write_op.cc:276] Timestamp: P: 1773782335873271 usec, L: 0
0317 21:19:07.531042 (+ 644us) write_op.cc:270] Start()
0317 21:19:07.531128 (+ 86us) write_op.cc:276] Timestamp: P: 1773782335891855 usec, L: 0
0317 21:19:07.531618 (+ 490us) write_op.cc:270] Start()
0317 21:19:07.531697 (+ 79us) write_op.cc:276] Timestamp: P: 1773782336005259 usec, L: 0
0317 21:19:07.532137 (+ 440us) write_op.cc:270] Start()
0317 21:19:07.532229 (+ 92us) write_op.cc:276] Timestamp: P: 1773782336007311 usec, L: 0
0317 21:19:07.541363 (+ 9134us) write_op.cc:270] Start()
0317 21:19:07.541480 (+ 117us) write_op.cc:276] Timestamp: P: 1773782336087759 usec, L: 0
0317 21:19:07.551539 (+ 10059us) write_op.cc:270] Start()
0317 21:19:07.551629 (+ 90us) write_op.cc:276] Timestamp: P: 1773782336092856 usec, L: 0
0317 21:19:07.555776 (+ 4147us) write_op.cc:270] Start()
0317 21:19:07.560127 (+ 4351us) write_op.cc:276] Timestamp: P: 1773782336132561 usec, L: 0
0317 21:19:07.560687 (+ 560us) write_op.cc:270] Start()
0317 21:19:07.560769 (+ 82us) write_op.cc:276] Timestamp: P: 1773782336144709 usec, L: 0
0317 21:19:07.562145 (+ 1376us) write_op.cc:270] Start()
0317 21:19:07.562229 (+ 84us) write_op.cc:276] Timestamp: P: 1773782336156207 usec, L: 0
0317 21:19:07.562702 (+ 473us) write_op.cc:270] Start()
0317 21:19:07.562785 (+ 83us) write_op.cc:276] Timestamp: P: 1773782336162724 usec, L: 0
0317 21:19:07.563723 (+ 938us) write_op.cc:270] Start()
0317 21:19:07.563807 (+ 84us) write_op.cc:276] Timestamp: P: 1773782336214315 usec, L: 0
0317 21:19:07.564254 (+ 447us) write_op.cc:270] Start()
0317 21:19:07.564322 (+ 68us) write_op.cc:276] Timestamp: P: 1773782336260440 usec, L: 0
0317 21:19:07.565931 (+ 1609us) write_op.cc:270] Start()
0317 21:19:07.566014 (+ 83us) write_op.cc:276] Timestamp: P: 1773782336371461 usec, L: 0
0317 21:19:07.566492 (+ 478us) write_op.cc:270] Start()
0317 21:19:07.566563 (+ 71us) write_op.cc:276] Timestamp: P: 1773782336449121 usec, L: 0
0317 21:19:07.567082 (+ 519us) write_op.cc:270] Start()
0317 21:19:07.567174 (+ 92us) write_op.cc:276] Timestamp: P: 1773782336623486 usec, L: 0
0317 21:19:07.567692 (+ 518us) write_op.cc:270] Start()
0317 21:19:07.567781 (+ 89us) write_op.cc:276] Timestamp: P: 1773782336654252 usec, L: 0
0317 21:19:07.568294 (+ 513us) write_op.cc:270] Start()
0317 21:19:07.568380 (+ 86us) write_op.cc:276] Timestamp: P: 1773782336656325 usec, L: 0
0317 21:19:07.568942 (+ 562us) write_op.cc:270] Start()
0317 21:19:07.569037 (+ 95us) write_op.cc:276] Timestamp: P: 1773782336658128 usec, L: 0
0317 21:19:07.569586 (+ 549us) write_op.cc:270] Start()
0317 21:19:07.569674 (+ 88us) write_op.cc:276] Timestamp: P: 1773782336780807 usec, L: 0
0317 21:19:07.572693 (+ 3019us) write_op.cc:270] Start()
0317 21:19:07.572907 (+ 214us) write_op.cc:276] Timestamp: P: 1773782336935315 usec, L: 0
0317 21:19:07.573468 (+ 561us) write_op.cc:270] Start()
0317 21:19:07.573554 (+ 86us) write_op.cc:276] Timestamp: P: 1773782336966460 usec, L: 0
0317 21:19:07.574376 (+ 822us) write_op.cc:270] Start()
0317 21:19:07.574462 (+ 86us) write_op.cc:276] Timestamp: P: 1773782336986176 usec, L: 0
0317 21:19:07.582021 (+ 7559us) write_op.cc:270] Start()
0317 21:19:07.582131 (+ 110us) write_op.cc:276] Timestamp: P: 1773782337281810 usec, L: 0
0317 21:19:07.585417 (+ 3286us) write_op.cc:270] Start()
0317 21:19:07.585510 (+ 93us) write_op.cc:276] Timestamp: P: 1773782337455534 usec, L: 0
0317 21:19:07.586062 (+ 552us) write_op.cc:270] Start()
0317 21:19:07.586139 (+ 77us) write_op.cc:276] Timestamp: P: 1773782337553330 usec, L: 0
0317 21:19:07.586605 (+ 466us) write_op.cc:270] Start()
0317 21:19:07.586683 (+ 78us) write_op.cc:276] Timestamp: P: 1773782337581756 usec, L: 0
0317 21:19:07.587173 (+ 490us) write_op.cc:270] Start()
0317 21:19:07.587252 (+ 79us) write_op.cc:276] Timestamp: P: 1773782337609996 usec, L: 0
0317 21:19:07.588133 (+ 881us) write_op.cc:270] Start()
0317 21:19:07.588210 (+ 77us) write_op.cc:276] Timestamp: P: 1773782337638375 usec, L: 0
0317 21:19:07.588699 (+ 489us) write_op.cc:270] Start()
0317 21:19:07.588768 (+ 69us) write_op.cc:276] Timestamp: P: 1773782337669299 usec, L: 0
0317 21:19:07.589203 (+ 435us) write_op.cc:270] Start()
0317 21:19:07.589295 (+ 92us) write_op.cc:276] Timestamp: P: 1773782337683867 usec, L: 0
0317 21:19:07.589828 (+ 533us) write_op.cc:270] Start()
0317 21:19:07.589905 (+ 77us) write_op.cc:276] Timestamp: P: 1773782337752557 usec, L: 0
0317 21:19:07.590343 (+ 438us) write_op.cc:270] Start()
0317 21:19:07.590422 (+ 79us) write_op.cc:276] Timestamp: P: 1773782337801969 usec, L: 0
0317 21:19:07.590888 (+ 466us) write_op.cc:270] Start()
0317 21:19:07.590955 (+ 67us) write_op.cc:276] Timestamp: P: 1773782337883566 usec, L: 0
0317 21:19:07.591389 (+ 434us) write_op.cc:270] Start()
0317 21:19:07.591464 (+ 75us) write_op.cc:276] Timestamp: P: 1773782338051599 usec, L: 0
0317 21:19:07.591886 (+ 422us) write_op.cc:270] Start()
0317 21:19:07.591954 (+ 68us) write_op.cc:276] Timestamp: P: 1773782338099881 usec, L: 0
0317 21:19:07.592371 (+ 417us) write_op.cc:270] Start()
0317 21:19:07.592438 (+ 67us) write_op.cc:276] Timestamp: P: 1773782338114083 usec, L: 0
0317 21:19:07.592896 (+ 458us) write_op.cc:270] Start()
0317 21:19:07.592963 (+ 67us) write_op.cc:276] Timestamp: P: 1773782338154142 usec, L: 0
0317 21:19:07.593371 (+ 408us) write_op.cc:270] Start()
0317 21:19:07.593438 (+ 67us) write_op.cc:276] Timestamp: P: 1773782338241135 usec, L: 0
0317 21:19:07.603089 (+ 9651us) write_op.cc:270] Start()
0317 21:19:07.603171 (+ 82us) write_op.cc:276] Timestamp: P: 1773782338263822 usec, L: 0
0317 21:19:07.603793 (+ 622us) write_op.cc:270] Start()
0317 21:19:07.603867 (+ 74us) write_op.cc:276] Timestamp: P: 1773782338310813 usec, L: 0
0317 21:19:07.606147 (+ 2280us) write_op.cc:270] Start()
0317 21:19:07.606223 (+ 76us) write_op.cc:276] Timestamp: P: 1773782338320358 usec, L: 0
0317 21:19:07.606665 (+ 442us) write_op.cc:270] Start()
0317 21:19:07.606729 (+ 64us) write_op.cc:276] Timestamp: P: 1773782338358581 usec, L: 0
0317 21:19:07.607131 (+ 402us) write_op.cc:270] Start()
0317 21:19:07.607198 (+ 67us) write_op.cc:276] Timestamp: P: 1773782338376344 usec, L: 0
0317 21:19:07.607600 (+ 402us) write_op.cc:270] Start()
0317 21:19:07.607680 (+ 80us) write_op.cc:276] Timestamp: P: 1773782338513218 usec, L: 0
0317 21:19:07.608239 (+ 559us) write_op.cc:270] Start()
0317 21:19:07.608308 (+ 69us) write_op.cc:276] Timestamp: P: 1773782338517348 usec, L: 0
0317 21:19:07.608808 (+ 500us) write_op.cc:270] Start()
0317 21:19:07.608877 (+ 69us) write_op.cc:276] Timestamp: P: 1773782338606027 usec, L: 0
0317 21:19:07.609281 (+ 404us) write_op.cc:270] Start()
0317 21:19:07.609350 (+ 69us) write_op.cc:276] Timestamp: P: 1773782338624646 usec, L: 0
0317 21:19:07.615426 (+ 6076us) write_op.cc:270] Start()
0317 21:19:07.615530 (+ 104us) write_op.cc:276] Timestamp: P: 1773782338645492 usec, L: 0
0317 21:19:07.616190 (+ 660us) write_op.cc:270] Start()
0317 21:19:07.616272 (+ 82us) write_op.cc:276] Timestamp: P: 1773782338647381 usec, L: 0
0317 21:19:07.616810 (+ 538us) write_op.cc:270] Start()
0317 21:19:07.616881 (+ 71us) write_op.cc:276] Timestamp: P: 1773782338673408 usec, L: 0
0317 21:19:07.617341 (+ 460us) write_op.cc:270] Start()
0317 21:19:07.617407 (+ 66us)
I20260317 21:19:13.140424 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 06f6a1c205a74045a9f79daf98809b45 (DEFAULT_TABLE table=default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.141520 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f6a1c205a74045a9f79daf98809b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.152915 7012 tablet_service.cc:1511] Processing CreateTablet for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6 (DEFAULT_TABLE table=default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.154497 7012 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.156281 7163 tablet_service.cc:1511] Processing CreateTablet for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6 (DEFAULT_TABLE table=default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.157728 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:13.161129 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 6deece45839849088dad3d873cc925b2 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6deece45839849088dad3d873cc925b2 has no consensus state
I20260317 21:19:13.180953 7008 tablet_service.cc:1511] Processing CreateTablet for tablet ae1797cc7fc346368c11ec68dcac3b2c (DEFAULT_TABLE table=default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.181254 7459 raft_consensus.cc:359] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.181885 7459 raft_consensus.cc:385] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.182132 7008 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1797cc7fc346368c11ec68dcac3b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.188220 7156 tablet_service.cc:1511] Processing CreateTablet for tablet ae1797cc7fc346368c11ec68dcac3b2c (DEFAULT_TABLE table=default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.189587 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1797cc7fc346368c11ec68dcac3b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.182135 7459 raft_consensus.cc:740] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.210250 7459 consensus_queue.cc:260] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.212685 7459 ts_tablet_manager.cc:1434] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.110s user 0.006s sys 0.000s
I20260317 21:19:13.243427 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:13.244024 7329 raft_consensus.cc:2468] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:13.247771 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 3130fb84aeed42c0bba58b970150c983 (DEFAULT_TABLE table=default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.249434 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 9531ac1602894c43b9c165d9a5eddc76 (DEFAULT_TABLE table=default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.249857 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3130fb84aeed42c0bba58b970150c983. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.249945 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6 (DEFAULT_TABLE table=default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.250494 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9531ac1602894c43b9c165d9a5eddc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.250928 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.251740 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 06f6a1c205a74045a9f79daf98809b45 (DEFAULT_TABLE table=default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.252615 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06f6a1c205a74045a9f79daf98809b45. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.255247 7164 tablet_service.cc:1511] Processing CreateTablet for tablet 5be44990b15345e1b827b8e954dd034d (DEFAULT_TABLE table=default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.256479 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be44990b15345e1b827b8e954dd034d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.261060 7151 tablet_service.cc:1511] Processing CreateTablet for tablet 2ccf1aa8fc9849978c752f1cebccb535 (DEFAULT_TABLE table=default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.262286 7151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ccf1aa8fc9849978c752f1cebccb535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.273662 8307 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:13.274354 8255 raft_consensus.cc:493] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.274879 8255 raft_consensus.cc:515] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.277310 8255 leader_election.cc:290] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.245872 7006 tablet_service.cc:1511] Processing CreateTablet for tablet 5be44990b15345e1b827b8e954dd034d (DEFAULT_TABLE table=default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.257973 7009 tablet_service.cc:1511] Processing CreateTablet for tablet 2ccf1aa8fc9849978c752f1cebccb535 (DEFAULT_TABLE table=default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.278395 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40549347f8463797d6791136883371" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:13.285408 7035 raft_consensus.cc:2468] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:13.279206 7462 tablet_bootstrap.cc:492] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:13.286331 7120 leader_election.cc:304] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:13.287089 8255 raft_consensus.cc:2804] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:13.287423 8255 raft_consensus.cc:493] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:13.287724 8255 raft_consensus.cc:3060] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.295356 7462 tablet_bootstrap.cc:654] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.295837 8255 raft_consensus.cc:515] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.282821 7007 tablet_service.cc:1511] Processing CreateTablet for tablet 9531ac1602894c43b9c165d9a5eddc76 (DEFAULT_TABLE table=default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.299275 7007 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9531ac1602894c43b9c165d9a5eddc76. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.301404 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d40549347f8463797d6791136883371" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:13.302066 7035 raft_consensus.cc:3060] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.313562 8255 leader_election.cc:290] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.314811 7035 raft_consensus.cc:2468] T 3d40549347f8463797d6791136883371 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:13.315910 7120 leader_election.cc:304] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:13.324843 8255 raft_consensus.cc:2804] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:13.325397 8255 raft_consensus.cc:697] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:13.326205 8255 consensus_queue.cc:237] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.329738 6973 leader_election.cc:304] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:13.330508 8253 raft_consensus.cc:2804] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:13.330835 8253 raft_consensus.cc:493] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:13.331118 8253 raft_consensus.cc:3060] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.340054 8253 raft_consensus.cc:515] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.343041 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:13.344631 6971 leader_election.cc:343] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.346791 8253 leader_election.cc:290] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.348462 7306 tablet_service.cc:1511] Processing CreateTablet for tablet ae1797cc7fc346368c11ec68dcac3b2c (DEFAULT_TABLE table=default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.350165 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 5be44990b15345e1b827b8e954dd034d (DEFAULT_TABLE table=default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.351763 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 2ccf1aa8fc9849978c752f1cebccb535 (DEFAULT_TABLE table=default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:13.352283 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:13.356091 7261 leader_election.cc:343] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.357805 8308 raft_consensus.cc:493] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.385599 7462 tablet_bootstrap.cc:492] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:13.386044 7462 ts_tablet_manager.cc:1403] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.107s user 0.010s sys 0.000s
I20260317 21:19:13.358283 8308 raft_consensus.cc:515] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.389623 8308 leader_election.cc:290] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:13.391171 7462 raft_consensus.cc:359] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.391829 7462 raft_consensus.cc:385] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.392077 7462 raft_consensus.cc:740] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.392658 7462 consensus_queue.cc:260] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.398767 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae1797cc7fc346368c11ec68dcac3b2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.400995 7462 ts_tablet_manager.cc:1434] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.015s user 0.005s sys 0.001s
I20260317 21:19:13.401249 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be44990b15345e1b827b8e954dd034d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.401891 7462 tablet_bootstrap.cc:492] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:13.402488 8289 raft_consensus.cc:493] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.403009 8289 raft_consensus.cc:515] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.405169 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ccf1aa8fc9849978c752f1cebccb535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.405296 8289 leader_election.cc:290] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.412568 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "440cb8b112044a53804357056f1226f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:13.414121 6971 leader_election.cc:343] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.423065 7462 tablet_bootstrap.cc:654] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.470134 7467 tablet_bootstrap.cc:492] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:13.470572 7467 ts_tablet_manager.cc:1403] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.454s user 0.011s sys 0.001s
I20260317 21:19:13.473258 7467 raft_consensus.cc:359] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.473886 7467 raft_consensus.cc:385] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.474128 7467 raft_consensus.cc:740] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.476191 7459 tablet_bootstrap.cc:492] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:13.487910 7459 tablet_bootstrap.cc:654] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.356716 7261 leader_election.cc:304] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:13.491717 8307 raft_consensus.cc:2749] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:13.492575 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 136514us
I20260317 21:19:13.494231 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:13.498756 7006 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5be44990b15345e1b827b8e954dd034d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.498756 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:13.501129 7009 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ccf1aa8fc9849978c752f1cebccb535. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:13.509189 7467 consensus_queue.cc:260] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.531698 7467 ts_tablet_manager.cc:1434] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.061s user 0.003s sys 0.004s
I20260317 21:19:13.532685 7467 tablet_bootstrap.cc:492] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:13.549382 7467 tablet_bootstrap.cc:654] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.561940 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:13.572305 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f9e98c10a83e429a98a0fb4328e1b0ee" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:13.572945 7329 raft_consensus.cc:3060] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.572834 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "440cb8b112044a53804357056f1226f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:13.573585 7326 raft_consensus.cc:2468] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:13.580065 8285 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 4 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.580530 8285 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:13.582438 8285 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:13.583885 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 5 candidate_status { last_received { term: 3 index: 421 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:13.584625 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 4 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 4.
I20260317 21:19:13.585075 7467 tablet_bootstrap.cc:492] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:13.585481 7467 ts_tablet_manager.cc:1403] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.000s
I20260317 21:19:13.586243 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:19:13.587057 7329 raft_consensus.cc:2468] T f9e98c10a83e429a98a0fb4328e1b0ee P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:13.588147 8285 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 4 FOLLOWER]: Leader pre-election won for term 5
I20260317 21:19:13.588222 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 5 candidate_status { last_received { term: 3 index: 421 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:13.588541 8285 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 4 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:13.589006 8285 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 4 FOLLOWER]: Advancing to term 5
I20260317 21:19:13.596462 8285 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:13.597504 7462 tablet_bootstrap.cc:492] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:13.597965 7462 ts_tablet_manager.cc:1403] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.196s user 0.009s sys 0.004s
I20260317 21:19:13.603087 6973 leader_election.cc:304] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:13.603956 8253 raft_consensus.cc:2804] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:13.604332 8253 raft_consensus.cc:493] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:13.604904 8253 raft_consensus.cc:3060] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:13.605238 7119 leader_election.cc:343] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3d40549347f8463797d6791136883371
I20260317 21:19:13.605433 7459 tablet_bootstrap.cc:492] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:13.605836 7459 ts_tablet_manager.cc:1403] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.130s user 0.005s sys 0.015s
I20260317 21:19:13.608100 7459 raft_consensus.cc:359] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.608925 7459 raft_consensus.cc:385] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 21:19:13.608903 7119 leader_election.cc:343] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 3d40549347f8463797d6791136883371
I20260317 21:19:13.609155 7459 raft_consensus.cc:740] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.608832 7467 raft_consensus.cc:359] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.609504 7467 raft_consensus.cc:385] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.609768 7467 raft_consensus.cc:740] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.609974 7459 consensus_queue.cc:260] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.610319 7467 consensus_queue.cc:260] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.612008 7459 ts_tablet_manager.cc:1434] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260317 21:19:13.612421 7467 ts_tablet_manager.cc:1434] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20260317 21:19:13.616349 8289 raft_consensus.cc:493] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.616909 8289 raft_consensus.cc:515] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.618789 8289 leader_election.cc:290] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.621037 8253 raft_consensus.cc:515] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.630774 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 5 candidate_status { last_received { term: 3 index: 421 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:19:13.630777 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 5 candidate_status { last_received { term: 3 index: 421 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
I20260317 21:19:13.631332 7918 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 4 FOLLOWER]: Advancing to term 5
W20260317 21:19:13.631682 7264 leader_election.cc:343] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.633363 8307 consensus_queue.cc:1048] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:13.635098 7264 leader_election.cc:343] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.620352 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "039d8bed256a43a6b98075b8430cbf3b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:13.638079 7180 raft_consensus.cc:2468] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:13.638423 7467 tablet_bootstrap.cc:492] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:13.641623 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 5.
I20260317 21:19:13.642819 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 5 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:13.653911 8255 raft_consensus.cc:493] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:13.654415 8255 raft_consensus.cc:515] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.656337 8255 leader_election.cc:290] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.657248 8285 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 5 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:13.658008 8285 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 FOLLOWER]: Leader election won for term 5
I20260317 21:19:13.658522 8285 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 LEADER]: Becoming Leader. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:13.659402 8285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 419, Committed index: 419, Last appended: 3.421, Last appended by leader: 421, Current term: 5, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:13.639487 6971 leader_election.cc:304] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:13.662901 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "440cb8b112044a53804357056f1226f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:13.664601 6971 leader_election.cc:343] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:13.664768 7261 leader_election.cc:343] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.665781 8253 leader_election.cc:290] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:13.666381 8253 raft_consensus.cc:2804] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:13.666497 7261 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:13.666836 8253 raft_consensus.cc:493] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:13.667219 8307 raft_consensus.cc:2749] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:13.668154 8253 raft_consensus.cc:3060] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.668489 8307 consensus_queue.cc:1048] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:13.674127 7459 tablet_bootstrap.cc:492] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:13.678272 7467 tablet_bootstrap.cc:654] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.679081 7462 raft_consensus.cc:359] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.679839 7462 raft_consensus.cc:385] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.680122 7462 raft_consensus.cc:740] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.680841 6973 leader_election.cc:304] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:13.680758 7462 consensus_queue.cc:260] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.683070 7918 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Rejecting Update request from peer 31726fd7f40442e49b2fe0f880793206 for earlier term 4. Current term is 5. Ops: []
I20260317 21:19:13.683143 7462 ts_tablet_manager.cc:1434] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.085s user 0.005s sys 0.000s
I20260317 21:19:13.683703 8289 raft_consensus.cc:2804] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:13.684183 8289 raft_consensus.cc:697] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:13.684528 8313 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 }, Status: INVALID_TERM, Last received: 0.0, Next index: 429, Last known committed idx: 308, Time since last communication: 0.000s
I20260317 21:19:13.685012 8289 consensus_queue.cc:237] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:13.688799 6737 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 46087.4us
I20260317 21:19:13.697754 8326 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 LEADER]: Stepping down as leader of term 4
I20260317 21:19:13.704394 6759 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 LEADER]: Rejecting Update request from peer 31726fd7f40442e49b2fe0f880793206 for earlier term 4. Current term is 5. Ops: []
I20260317 21:19:13.714148 8325 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
W20260317 21:19:13.712407 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 44102.2us
W20260317 21:19:13.718894 7261 leader_election.cc:343] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:13.719411 7261 leader_election.cc:304] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:13.720839 7462 tablet_bootstrap.cc:492] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:13.718487 8292 op_tracker.cc:284] OpTracker waiting for 1 outstanding ops to complete now for 1.079s
I20260317 21:19:13.724982 8292 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20260317 21:19:13.725584 8326 raft_consensus.cc:740] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 LEADER]: Becoming Follower/Learner. State: Replica: 31726fd7f40442e49b2fe0f880793206, State: Running, Role: LEADER
I20260317 21:19:13.707296 8313 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 }, Status: INVALID_TERM, Last received: 0.0, Next index: 429, Last known committed idx: 419, Time since last communication: 0.001s
I20260317 21:19:13.725368 8292 op_tracker.cc:290] R-P WriteOp [type=FOLLOWER, start_time=2026-03-17 21:19:12, state=WriteOpState 0x7b5800083700 [op_id=(term: 2 index: 420), ts=7265412512810127360, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>)]]]
I20260317 21:19:13.726433 8313 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [LEADER]: Peer 21186091f2c9410ea6e585c5ef2c6291 log is divergent from this leader: its last log entry 5.422 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 419
I20260317 21:19:13.727115 8325 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:13.728955 8307 raft_consensus.cc:2749] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:13.732463 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "039d8bed256a43a6b98075b8430cbf3b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:13.733318 7328 raft_consensus.cc:2468] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:13.734364 8326 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 420, Committed index: 420, Last appended: 4.429, Last appended by leader: 429, Current term: 4, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:13.735610 8326 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 4 FOLLOWER]: Advancing to term 5
W20260317 21:19:13.736760 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 55983.7us
I20260317 21:19:13.738237 7467 tablet_bootstrap.cc:492] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:13.739826 7459 tablet_bootstrap.cc:654] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.763751 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:13.764559 7035 raft_consensus.cc:2393] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:13.771730 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "440cb8b112044a53804357056f1226f1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:13.772258 7328 raft_consensus.cc:3060] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.779168 7328 raft_consensus.cc:2468] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:13.782173 7462 tablet_bootstrap.cc:654] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:13.787421 8324 consensus_queue.cc:1048] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.009s
I20260317 21:19:13.798367 7462 tablet_bootstrap.cc:492] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:13.798857 7462 ts_tablet_manager.cc:1403] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.000s
I20260317 21:19:13.801368 7462 raft_consensus.cc:359] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.802253 7462 raft_consensus.cc:385] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.802551 7462 raft_consensus.cc:740] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.803253 7462 consensus_queue.cc:260] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.819832 7462 ts_tablet_manager.cc:1434] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.021s user 0.005s sys 0.000s
I20260317 21:19:13.832037 7467 ts_tablet_manager.cc:1403] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.193s user 0.013s sys 0.000s
W20260317 21:19:13.842445 7119 leader_election.cc:343] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: c9b7f7c864004b9abde0a0201e196165
I20260317 21:19:13.879302 7467 raft_consensus.cc:359] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.884269 7467 raft_consensus.cc:385] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.888988 7467 raft_consensus.cc:740] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.898440 7462 tablet_bootstrap.cc:492] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:13.933961 7462 tablet_bootstrap.cc:654] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:13.936133 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 157629us
I20260317 21:19:13.937682 6973 leader_election.cc:304] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:13.943534 8324 raft_consensus.cc:2804] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:13.944110 8324 raft_consensus.cc:697] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:13.974054 8253 raft_consensus.cc:515] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.978056 7459 tablet_bootstrap.cc:492] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:13.978494 7459 ts_tablet_manager.cc:1403] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.305s user 0.009s sys 0.007s
I20260317 21:19:13.983644 7459 raft_consensus.cc:359] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.984302 7459 raft_consensus.cc:385] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:13.989352 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "039d8bed256a43a6b98075b8430cbf3b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:13.990022 7328 raft_consensus.cc:3060] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:13.990072 8324 consensus_queue.cc:237] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:13.996098 7459 raft_consensus.cc:740] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:13.976562 8253 leader_election.cc:290] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:14.014132 7459 consensus_queue.cc:260] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.017033 7459 ts_tablet_manager.cc:1434] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.038s user 0.008s sys 0.001s
I20260317 21:19:14.044927 7328 raft_consensus.cc:2468] T 039d8bed256a43a6b98075b8430cbf3b P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:14.049477 7459 tablet_bootstrap.cc:492] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:14.046125 6973 leader_election.cc:304] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:14.050450 8324 raft_consensus.cc:2804] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:14.051200 8324 raft_consensus.cc:697] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:14.052073 8324 consensus_queue.cc:237] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.130501 8307 raft_consensus.cc:493] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.130977 8307 raft_consensus.cc:515] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.129858 7467 consensus_queue.cc:260] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.141443 7467 ts_tablet_manager.cc:1434] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.308s user 0.006s sys 0.003s
I20260317 21:19:14.157377 8307 leader_election.cc:290] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:14.158712 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "252f728d183f440a9ef42894508d5755" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:14.159303 7035 raft_consensus.cc:2468] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:14.160722 7264 leader_election.cc:304] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:14.162846 8307 raft_consensus.cc:2804] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:14.163317 8307 raft_consensus.cc:493] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:14.163718 8307 raft_consensus.cc:3060] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.169205 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "252f728d183f440a9ef42894508d5755" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:14.129649 7119 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.6.122.126:40107, user_credentials={real_user=slave}} blocked reactor thread for 221934us
W20260317 21:19:14.170864 7261 leader_election.cc:343] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:14.174788 7182 raft_consensus.cc:3060] T 2fd1d8d70e9849fa9e107b2dd3355c58 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.177410 7184 raft_consensus.cc:3060] T 564d41fc13b04ec688fc38a78f55e83d P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.181996 7467 tablet_bootstrap.cc:492] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:14.185070 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "039d8bed256a43a6b98075b8430cbf3b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:14.185285 7119 consensus_peers.cc:597] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc8567ae10464694a1128ba2eb1d8ee1. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:14.185878 7179 raft_consensus.cc:3060] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.185984 8339 consensus_queue.cc:1048] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:14.189396 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 6deece45839849088dad3d873cc925b2 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6deece45839849088dad3d873cc925b2 has no consensus state
I20260317 21:19:14.201954 6683 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Refusing update from remote peer 21186091f2c9410ea6e585c5ef2c6291: Log matching property violated. Preceding OpId in replica: term: 4 index: 429. Preceding OpId from leader: term: 5 index: 422. (term mismatch)
I20260317 21:19:14.202386 6683 pending_rounds.cc:85] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting all ops after (but not including) 421
I20260317 21:19:14.202687 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.422
I20260317 21:19:14.203652 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.423
W20260317 21:19:14.203774 6666 catalog_manager.cc:6209] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: An error occurred while updating sys-tables: Aborted: Op aborted by new leader
W20260317 21:19:14.204869 6668 catalog_manager.cc:2202] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260317 21:19:14.206027 6668 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39038 (request call id 390) took 2849 ms. Trace:
I20260317 21:19:14.206336 6668 rpcz_store.cc:276] 0317 21:19:11.356209 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.451577 (+ 95368us) service_pool.cc:225] Handling call
0317 21:19:11.490494 (+ 38917us) spinlock_profiling.cc:243] Waited 36.9 ms on lock 0x7b5400030554. stack: 00007fc0553ba423 00007fc0553ba320 00007fc054f2f574 0000000000509544 00000000005094d9 00007fc05c4162c0 00007fc05c4b4e6a 00007fc05e1b721c 00007fc05e2b1076 00007fc05bed2524 00007fc05bed24b2 00007fc05bed23e1 00007fc05bed235c 00007fc05bed1612 00007fc05a1a834c 00007fc05a1a7786
0317 21:19:11.702858 (+212364us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:12.109448 (+406590us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:14.205942 (+2096494us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.203599 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.203964 (+ 365us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.203984 (+ 20us) write_op.cc:435] Acquired schema lock
0317 21:19:12.203996 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:12.204235 (+ 239us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.204268 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.204288 (+ 20us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:12.204461 (+ 173us) tablet.cc:700] Row locks acquired
0317 21:19:12.204492 (+ 31us) write_op.cc:260] PREPARE: finished
0317 21:19:12.204602 (+ 110us) write_op.cc:270] Start()
0317 21:19:12.206104 (+ 1502us) write_op.cc:276] Timestamp: P: 1773782352204476 usec, L: 0
0317 21:19:12.206122 (+ 18us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.206582 (+ 460us) log.cc:844] Serialized 433 byte log entry
0317 21:19:14.204061 (+1997479us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.204099 (+ 38us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.204135 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.204400 (+ 265us) write_op.cc:454] Released schema lock
0317 21:19:14.204441 (+ 41us) write_op.cc:335] FINISH: Op aborted
Metrics: {"spinlock_wait_cycles":84864256,"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":91926,"prepare.run_cpu_time_us":1828,"prepare.run_wall_time_us":4013,"replication_time_us":1997803}]]}
W20260317 21:19:14.211316 7386 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.126:40107)
I20260317 21:19:14.209630 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.424
I20260317 21:19:14.213155 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.425
W20260317 21:19:14.214314 6674 catalog_manager.cc:2848] Aborted: an error occurred while updating the sys-catalog: Op aborted by new leader
W20260317 21:19:14.213165 6672 catalog_manager.cc:2202] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260317 21:19:14.215175 6674 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 394) took 2560 ms. Trace:
I20260317 21:19:14.215545 6672 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39038 (request call id 392) took 2790 ms. Trace:
I20260317 21:19:14.215538 6674 rpcz_store.cc:276] 0317 21:19:11.655178 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:12.153157 (+497979us) service_pool.cc:225] Handling call
0317 21:19:12.154913 (+ 1756us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:12.173596 (+ 18683us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:12.173852 (+ 256us) catalog_manager.cc:2824] Locking tablets
0317 21:19:12.174541 (+ 689us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:14.215084 (+2040543us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.211395 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.211871 (+ 476us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.211895 (+ 24us) write_op.cc:435] Acquired schema lock
0317 21:19:12.211908 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:12.212195 (+ 287us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.212228 (+ 33us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.212250 (+ 22us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:12.212491 (+ 241us) tablet.cc:700] Row locks acquired
0317 21:19:12.212503 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:12.212614 (+ 111us) write_op.cc:270] Start()
0317 21:19:12.212713 (+ 99us) write_op.cc:276] Timestamp: P: 1773782352212490 usec, L: 0
0317 21:19:12.212729 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.213167 (+ 438us) log.cc:844] Serialized 728 byte log entry
0317 21:19:14.213682 (+2000515us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.213753 (+ 71us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.213781 (+ 28us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.213918 (+ 137us) write_op.cc:454] Released schema lock
0317 21:19:14.213955 (+ 37us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":34071,"prepare.run_cpu_time_us":1878,"prepare.run_wall_time_us":2865,"replication_time_us":2000817}]]}
I20260317 21:19:14.215955 6672 rpcz_store.cc:276] 0317 21:19:11.424896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:12.117246 (+692350us) service_pool.cc:225] Handling call
0317 21:19:12.131018 (+ 13772us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:12.132781 (+ 1763us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:14.215477 (+2082696us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.207786 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.208170 (+ 384us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.208191 (+ 21us) write_op.cc:435] Acquired schema lock
0317 21:19:12.208203 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:12.208466 (+ 263us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.208502 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.208534 (+ 32us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:12.209257 (+ 723us) tablet.cc:700] Row locks acquired
0317 21:19:12.209271 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:12.209379 (+ 108us) write_op.cc:270] Start()
0317 21:19:12.209460 (+ 81us) write_op.cc:276] Timestamp: P: 1773782352209253 usec, L: 0
0317 21:19:12.209476 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.209954 (+ 478us) log.cc:844] Serialized 433 byte log entry
0317 21:19:14.211960 (+2002006us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.212024 (+ 64us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.212053 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.212217 (+ 164us) write_op.cc:454] Released schema lock
0317 21:19:14.212254 (+ 37us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":73456,"prepare.run_cpu_time_us":1834,"prepare.run_wall_time_us":3483,"replication_time_us":2002330}]]}
W20260317 21:19:14.220477 7386 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.122.126:40107)
I20260317 21:19:14.216279 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.426
W20260317 21:19:14.223232 6675 catalog_manager.cc:6209] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: An error occurred while updating sys-tables: Aborted: Op aborted by new leader
W20260317 21:19:14.226462 6666 catalog_manager.cc:6209] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: An error occurred while updating sys-tables: Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER
W20260317 21:19:14.226953 6675 catalog_manager.cc:6209] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: An error occurred while updating sys-tables: Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER
W20260317 21:19:14.229496 6666 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:45181 (request call id 70) took 2992 ms (client timeout 1000 ms). Trace:
W20260317 21:19:14.229832 6666 rpcz_store.cc:269] 0317 21:19:11.237307 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.243332 (+ 6025us) service_pool.cc:225] Handling call
0317 21:19:14.226199 (+2982867us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.226226 (+ 27us) write_op.cc:454] Released schema lock
0317 21:19:14.229417 (+ 3191us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.201029 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.201479 (+ 450us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.201539 (+ 60us) write_op.cc:435] Acquired schema lock
0317 21:19:12.201555 (+ 16us) tablet.cc:661] Decoding operations
0317 21:19:12.201900 (+ 345us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.201951 (+ 51us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.201971 (+ 20us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:12.202147 (+ 176us) tablet.cc:700] Row locks acquired
0317 21:19:12.202163 (+ 16us) write_op.cc:260] PREPARE: finished
0317 21:19:12.202311 (+ 148us) write_op.cc:270] Start()
0317 21:19:12.202428 (+ 117us) write_op.cc:276] Timestamp: P: 1773782352202173 usec, L: 0
0317 21:19:12.202451 (+ 23us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.203167 (+ 716us) log.cc:844] Serialized 356 byte log entry
0317 21:19:14.203154 (+1999987us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.203216 (+ 62us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.203243 (+ 27us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.203397 (+ 154us) write_op.cc:454] Released schema lock
0317 21:19:14.203432 (+ 35us) write_op.cc:335] FINISH: Op aborted
Related trace 'op':
0317 21:19:14.225942 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER)
0317 21:19:14.225989 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.226018 (+ 29us) write_op.cc:454] Released schema lock
0317 21:19:14.226053 (+ 35us) write_op.cc:335] FINISH: Op aborted
Metrics: {"reported_tablets":9,"child_traces":[["op",{"mutex_wait_us":430788,"num_ops":1,"prepare.queue_time_us":523954,"prepare.run_cpu_time_us":2445,"prepare.run_wall_time_us":2444,"replication_time_us":2000539}],["op",{}]]}
I20260317 21:19:14.233959 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.427
I20260317 21:19:14.212227 8307 raft_consensus.cc:515] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:14.244768 6673 catalog_manager.cc:2202] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260317 21:19:14.245695 6673 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:39038 (request call id 393) took 2779 ms. Trace:
I20260317 21:19:14.246073 6673 rpcz_store.cc:276] 0317 21:19:11.466163 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:12.151969 (+685806us) service_pool.cc:225] Handling call
0317 21:19:12.260467 (+108498us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:12.262016 (+ 1549us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:14.245520 (+1983504us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.272967 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.273585 (+ 618us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.273608 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:19:12.273620 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:12.273910 (+ 290us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.273947 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.273967 (+ 20us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:12.274161 (+ 194us) tablet.cc:700] Row locks acquired
0317 21:19:12.274174 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:12.274303 (+ 129us) write_op.cc:270] Start()
0317 21:19:12.274381 (+ 78us) write_op.cc:276] Timestamp: P: 1773782352274162 usec, L: 0
0317 21:19:12.274395 (+ 14us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.274963 (+ 568us) log.cc:844] Serialized 433 byte log entry
0317 21:19:14.240992 (+1966029us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.241047 (+ 55us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.241088 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.241249 (+ 161us) write_op.cc:454] Released schema lock
0317 21:19:14.241289 (+ 40us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":9660,"prepare.run_cpu_time_us":2064,"prepare.run_wall_time_us":3115,"replication_time_us":1966313}]]}
I20260317 21:19:14.251983 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.428
I20260317 21:19:14.252135 7918 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Refusing update from remote peer 21186091f2c9410ea6e585c5ef2c6291: Log matching property violated. Preceding OpId in replica: term: 2 index: 312. Preceding OpId from leader: term: 5 index: 422. (index mismatch)
I20260317 21:19:14.253950 8325 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 422, Last known committed idx: 308, Time since last communication: 0.001s
I20260317 21:19:14.256618 7462 tablet_bootstrap.cc:492] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:14.257092 7462 ts_tablet_manager.cc:1403] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.359s user 0.009s sys 0.022s
I20260317 21:19:14.257218 8307 leader_election.cc:290] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:14.257681 8307 raft_consensus.cc:493] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.258087 8307 raft_consensus.cc:515] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:14.260725 6670 catalog_manager.cc:2848] Aborted: an error occurred while updating the sys-catalog: Op aborted by new leader
I20260317 21:19:14.261356 6670 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:39038 (request call id 395) took 2605 ms. Trace:
I20260317 21:19:14.261667 6670 rpcz_store.cc:276] 0317 21:19:11.655445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:12.263568 (+608123us) service_pool.cc:225] Handling call
0317 21:19:12.265009 (+ 1441us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:12.265731 (+ 722us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:12.265904 (+ 173us) catalog_manager.cc:2824] Locking tablets
0317 21:19:12.266439 (+ 535us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:14.261294 (+1994855us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.276216 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.276581 (+ 365us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.276606 (+ 25us) write_op.cc:435] Acquired schema lock
0317 21:19:12.276619 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:12.276961 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.276997 (+ 36us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.277019 (+ 22us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:12.277291 (+ 272us) tablet.cc:700] Row locks acquired
0317 21:19:12.277304 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:12.277410 (+ 106us) write_op.cc:270] Start()
0317 21:19:12.277487 (+ 77us) write_op.cc:276] Timestamp: P: 1773782352277278 usec, L: 0
0317 21:19:12.277501 (+ 14us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.278122 (+ 621us) log.cc:844] Serialized 728 byte log entry
0317 21:19:14.254577 (+1976455us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.254623 (+ 46us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.254663 (+ 40us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.254940 (+ 277us) write_op.cc:454] Released schema lock
0317 21:19:14.254985 (+ 45us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"num_ops":2,"prepare.queue_time_us":7891,"prepare.run_cpu_time_us":2063,"prepare.run_wall_time_us":44566,"replication_time_us":1976887}]]}
I20260317 21:19:14.265442 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted NO_OP operation due to leader change: 4.429
I20260317 21:19:14.272104 6683 raft_consensus.cc:2889] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260317 21:19:14.275034 8307 leader_election.cc:290] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:14.253687 6675 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:36209 (request call id 62) took 3018 ms (client timeout 1000 ms). Trace:
W20260317 21:19:14.283601 6675 rpcz_store.cc:269] 0317 21:19:11.235099 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.236037 (+ 938us) service_pool.cc:225] Handling call
0317 21:19:14.226524 (+2990487us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.226552 (+ 28us) write_op.cc:454] Released schema lock
0317 21:19:14.253605 (+ 27053us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:12.214378 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:12.214755 (+ 377us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:12.214782 (+ 27us) write_op.cc:435] Acquired schema lock
0317 21:19:12.214804 (+ 22us) tablet.cc:661] Decoding operations
0317 21:19:12.215052 (+ 248us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:12.215087 (+ 35us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:12.215107 (+ 20us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:12.215231 (+ 124us) tablet.cc:700] Row locks acquired
0317 21:19:12.215245 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:12.215341 (+ 96us) write_op.cc:270] Start()
0317 21:19:12.215410 (+ 69us) write_op.cc:276] Timestamp: P: 1773782352215216 usec, L: 0
0317 21:19:12.215425 (+ 15us) op_driver.cc:348] REPLICATION: starting
0317 21:19:12.215864 (+ 439us) log.cc:844] Serialized 356 byte log entry
0317 21:19:14.222436 (+2006572us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.222487 (+ 51us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.222530 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.222689 (+ 159us) write_op.cc:454] Released schema lock
0317 21:19:14.222733 (+ 44us) write_op.cc:335] FINISH: Op aborted
Related trace 'op':
0317 21:19:14.226315 (+ 0us) op_driver.cc:389] HandleFailure(Illegal state: replica 31726fd7f40442e49b2fe0f880793206 is not leader of this config: current role FOLLOWER)
0317 21:19:14.226351 (+ 36us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.226375 (+ 24us) write_op.cc:454] Released schema lock
0317 21:19:14.226398 (+ 23us) write_op.cc:335] FINISH: Op aborted
Metrics: {"mutex_wait_us":537055,"reported_tablets":15,"spinlock_wait_cycles":17664,"child_traces":[["op",{"num_ops":1,"prepare.queue_time_us":30579,"prepare.run_cpu_time_us":1667,"prepare.run_wall_time_us":2499,"replication_time_us":2006826}],["op",{}]]}
I20260317 21:19:14.308073 8353 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 422, Last known committed idx: 420, Time since last communication: 0.001s
I20260317 21:19:14.317791 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "252f728d183f440a9ef42894508d5755" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:14.318337 7036 raft_consensus.cc:3060] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.319953 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 with delete_type TABLET_DATA_DELETED (replaced by tablet 6deece45839849088dad3d873cc925b2) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:14.321189 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:14.322410 7305 tablet_service.cc:1558] Processing DeleteTablet for tablet 57e721ac4b24438b9b2a0e2569027236 with delete_type TABLET_DATA_DELETED (Sending initial creation of tablet) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:14.323174 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet c9b7f7c864004b9abde0a0201e196165 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:14.331472 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "252f728d183f440a9ef42894508d5755" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:14.334156 7261 leader_election.cc:343] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:14.334390 7467 tablet_bootstrap.cc:654] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:14.335547 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:14.337036 7261 leader_election.cc:343] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:14.330345 7036 raft_consensus.cc:2468] T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:14.351481 8339 raft_consensus.cc:493] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.352100 8339 raft_consensus.cc:515] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.354605 8339 leader_election.cc:290] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:14.357300 6973 leader_election.cc:343] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: c9b7f7c864004b9abde0a0201e196165
I20260317 21:19:14.359165 7179 raft_consensus.cc:2468] T 039d8bed256a43a6b98075b8430cbf3b P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:14.365248 8339 consensus_queue.cc:1048] T 2fd1d8d70e9849fa9e107b2dd3355c58 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:14.389071 7151 tablet_service.cc:1558] Processing DeleteTablet for tablet 3e2f891e712341868d94bab1dd72a9e1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:14.413662 8339 consensus_queue.cc:1048] T 564d41fc13b04ec688fc38a78f55e83d P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:14.421324 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet 7eb432b6bccf4e32a2fa9d4979586c52 with delete_type TABLET_DATA_DELETED (replaced by tablet 6deece45839849088dad3d873cc925b2) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:14.422263 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 5314c562bc9d457a8e1f4930e8d3a7ff with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:14.423642 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet bcd99ca9f6c540e48e29d8659fbfe777 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:57 UTC) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:14.440172 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b066a6a5c864beb8da59bd97bacd309 with delete_type TABLET_DATA_DELETED (replaced by tablet 4f33574134934f048632aae269244368) from {username='slave'} at 127.0.0.1:42830
I20260317 21:19:14.440724 7120 leader_election.cc:304] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:14.442711 8255 raft_consensus.cc:3060] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.445158 8359 tablet_replica.cc:333] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:14.446205 8359 raft_consensus.cc:2243] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:14.447077 7459 tablet_bootstrap.cc:654] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:14.449806 8256 consensus_queue.cc:1048] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:14.451171 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:14.456880 7462 raft_consensus.cc:359] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.287779 8326 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.461563 8326 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
W20260317 21:19:14.309427 8314 op_tracker.cc:284] OpTracker waiting for 1 outstanding ops to complete now for 1.076s
I20260317 21:19:14.462839 8314 op_tracker.cc:286] Dumping up to 50 currently running ops:
I20260317 21:19:14.463718 8326 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:14.463171 8314 op_tracker.cc:290] R-P WriteOp [type=LEADER, start_time=2026-03-17 21:19:11, state=WriteOpState 0x7b5800135f00 [op_id=(), ts=7265412512827645952, rows=[INSERT (int8 entry_type=<redacted>, string entry_id=<redacted>, string metadata=<redacted>), MUTATE (int8 entry_type=<redacted>, string entry_id=<redacted>) SET metadata=<redacted>]]]
I20260317 21:19:14.467778 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 2 index: 421 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:14.470054 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 2 index: 421 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:14.475507 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:14.476799 8326 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 21:19:14.309775 8353 consensus_queue.cc:1243] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Peer 31726fd7f40442e49b2fe0f880793206 log is divergent from this leader: its last log entry 2.421 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 420
I20260317 21:19:14.493428 6683 pending_rounds.cc:85] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting all ops after (but not including) 420
I20260317 21:19:14.493785 6683 pending_rounds.cc:107] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206: Aborting uncommitted WRITE_OP operation due to leader change: 2.421
W20260317 21:19:14.496315 6669 catalog_manager.cc:6706] Aborted: an error occurred while writing to the sys-catalog: Op aborted by new leader
I20260317 21:19:14.497285 6669 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:39038 (request call id 391) took 3100 ms. Trace:
I20260317 21:19:14.497994 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet 440cb8b112044a53804357056f1226f1 with delete_type TABLET_DATA_DELETED (replaced by tablet b35816e1de7d4b13927848538cdc3e78) from {username='slave'} at 127.0.0.1:34096
I20260317 21:19:14.497605 6669 rpcz_store.cc:276] 0317 21:19:11.397146 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:11.547370 (+150224us) service_pool.cc:225] Handling call
0317 21:19:14.497211 (+2949841us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:11.762130 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:11.762689 (+ 559us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:11.762715 (+ 26us) write_op.cc:435] Acquired schema lock
0317 21:19:11.762729 (+ 14us) tablet.cc:661] Decoding operations
0317 21:19:11.764248 (+ 1519us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:11.764285 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:11.764308 (+ 23us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:11.764517 (+ 209us) tablet.cc:700] Row locks acquired
0317 21:19:11.764530 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:11.764679 (+ 149us) write_op.cc:270] Start()
0317 21:19:11.764763 (+ 84us) write_op.cc:276] Timestamp: P: 1773782351764562 usec, L: 0
0317 21:19:11.764779 (+ 16us) op_driver.cc:348] REPLICATION: starting
0317 21:19:11.767210 (+ 2431us) log.cc:844] Serialized 625 byte log entry
0317 21:19:14.494155 (+2726945us) op_driver.cc:464] REPLICATION: finished
0317 21:19:14.494221 (+ 66us) op_driver.cc:389] HandleFailure(Aborted: Op aborted by new leader)
0317 21:19:14.494253 (+ 32us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:14.494389 (+ 136us) write_op.cc:454] Released schema lock
0317 21:19:14.494422 (+ 33us) write_op.cc:335] FINISH: Op aborted
Metrics: {"child_traces":[["op",{"mutex_wait_us":49206,"num_ops":2,"prepare.queue_time_us":52191,"prepare.run_cpu_time_us":2468,"prepare.run_wall_time_us":6737,"replication_time_us":2729237}]]}
W20260317 21:19:14.500964 7386 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.122.126:40107)
I20260317 21:19:14.508240 8354 tablet_replica.cc:333] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:14.509961 8354 raft_consensus.cc:2243] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:14.510558 8354 raft_consensus.cc:2272] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:14.513049 8354 ts_tablet_manager.cc:1916] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:14.457782 7462 raft_consensus.cc:385] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:14.529182 7462 raft_consensus.cc:740] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:14.530097 7462 consensus_queue.cc:260] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.532497 8326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:14.533284 8326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
W20260317 21:19:14.538724 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 7eb432b6bccf4e32a2fa9d4979586c52 because the tablet was not found. No further retry: Not found: Tablet not found: 7eb432b6bccf4e32a2fa9d4979586c52
W20260317 21:19:14.540341 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 57e721ac4b24438b9b2a0e2569027236 because the tablet was not found. No further retry: Not found: Tablet not found: 57e721ac4b24438b9b2a0e2569027236
I20260317 21:19:14.557193 8357 raft_consensus.cc:493] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.557798 8357 raft_consensus.cc:515] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.562129 8357 leader_election.cc:290] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:14.569587 8326 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:14.570295 8326 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:14.574435 8373 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:14.574435 8359 pending_rounds.cc:70] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:19:14.578868 8359 pending_rounds.cc:77] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412513582940160 op_type: NO_OP noop_request { }
I20260317 21:19:14.579538 8373 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
W20260317 21:19:14.581589 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 138928us
I20260317 21:19:14.588737 8359 raft_consensus.cc:2889] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:14.603718 8337 raft_consensus.cc:493] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.604337 8337 raft_consensus.cc:515] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.607193 8337 leader_election.cc:290] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:14.609001 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:14.609782 7328 raft_consensus.cc:2468] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:14.612635 6973 leader_election.cc:304] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:14.613517 8337 raft_consensus.cc:2804] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:14.614114 8337 raft_consensus.cc:493] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:14.614760 8337 raft_consensus.cc:3060] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.597275 8359 raft_consensus.cc:2272] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:14.625831 8256 consensus_queue.cc:1048] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:14.627727 7120 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 5b066a6a5c864beb8da59bd97bacd309
W20260317 21:19:14.640103 7119 consensus_peers.cc:597] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d40549347f8463797d6791136883371. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:14.642692 8351 raft_consensus.cc:493] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.643368 8351 raft_consensus.cc:515] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.647023 8351 leader_election.cc:290] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:14.666826 8314 catalog_manager.cc:1432] Term changed from 4 to 5 while waiting for master leader catchup. Not loading sys catalog metadata
I20260317 21:19:14.669262 7462 ts_tablet_manager.cc:1434] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.412s user 0.004s sys 0.004s
I20260317 21:19:14.670433 7462 tablet_bootstrap.cc:492] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:14.698630 7459 tablet_bootstrap.cc:492] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:14.699208 7459 ts_tablet_manager.cc:1403] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.650s user 0.009s sys 0.002s
W20260317 21:19:14.623222 6661 catalog_manager.cc:4969] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet c9b7f7c864004b9abde0a0201e196165 because the tablet was not found. No further retry: Not found: Tablet not found: c9b7f7c864004b9abde0a0201e196165
W20260317 21:19:14.704698 6661 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 81516.6us
W20260317 21:19:14.720268 7264 leader_election.cc:343] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:14.729843 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:14.731642 8255 raft_consensus.cc:2749] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:14.738426 7180 raft_consensus.cc:2468] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:14.735680 7261 leader_election.cc:343] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:14.743546 7264 leader_election.cc:304] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:14.745129 7264 leader_election.cc:304] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:14.746079 8351 raft_consensus.cc:2749] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:14.747654 8352 raft_consensus.cc:2804] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:14.760403 8359 ts_tablet_manager.cc:1916] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:14.774839 7467 tablet_bootstrap.cc:492] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:14.775306 7467 ts_tablet_manager.cc:1403] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.594s user 0.013s sys 0.005s
I20260317 21:19:14.776288 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:14.777652 7467 raft_consensus.cc:359] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.778265 7467 raft_consensus.cc:385] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:14.778522 7467 raft_consensus.cc:740] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:14.779069 7467 consensus_queue.cc:260] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.785394 6971 leader_election.cc:304] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
W20260317 21:19:14.789842 6971 consensus_peers.cc:597] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:14.790386 6971 leader_election.cc:343] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:14.808004 8340 raft_consensus.cc:2804] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 21:19:14.808528 8340 raft_consensus.cc:493] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:14.810918 8340 raft_consensus.cc:3060] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:14.824079 8339 consensus_queue.cc:1048] T f9e98c10a83e429a98a0fb4328e1b0ee P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:14.826054 7467 ts_tablet_manager.cc:1434] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.050s user 0.004s sys 0.000s
I20260317 21:19:14.827364 7467 tablet_bootstrap.cc:492] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
W20260317 21:19:14.837849 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 46693.1us
I20260317 21:19:14.848136 8337 raft_consensus.cc:515] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.871354 7467 tablet_bootstrap.cc:654] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:14.905210 8368 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:14.906054 8368 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:14.913344 8352 raft_consensus.cc:697] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:14.914225 8352 consensus_queue.cc:237] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.923645 8307 raft_consensus.cc:493] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:14.924114 8307 raft_consensus.cc:515] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.926034 7467 tablet_bootstrap.cc:492] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:14.925938 8307 leader_election.cc:290] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:14.926420 7467 ts_tablet_manager.cc:1403] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.099s user 0.006s sys 0.003s
I20260317 21:19:14.929317 7467 raft_consensus.cc:359] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.929973 7467 raft_consensus.cc:385] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:14.930218 7467 raft_consensus.cc:740] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:14.930773 7467 consensus_queue.cc:260] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.932385 7467 ts_tablet_manager.cc:1434] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20260317 21:19:14.933341 7467 tablet_bootstrap.cc:492] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:14.941818 8359 ts_tablet_manager.cc:1929] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:14.942303 8359 log.cc:1199] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/3e2f891e712341868d94bab1dd72a9e1
I20260317 21:19:14.943241 8359 ts_tablet_manager.cc:1950] T 3e2f891e712341868d94bab1dd72a9e1 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:14.907956 7459 raft_consensus.cc:359] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.951305 7459 raft_consensus.cc:385] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:14.951884 8337 leader_election.cc:290] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:14.951670 7459 raft_consensus.cc:740] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:14.953253 7459 consensus_queue.cc:260] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.917354 8340 raft_consensus.cc:515] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:14.958748 8340 leader_election.cc:290] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:14.962481 7467 tablet_bootstrap.cc:654] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:14.948592 8354 ts_tablet_manager.cc:1929] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:14.967553 8354 log.cc:1199] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/440cb8b112044a53804357056f1226f1
I20260317 21:19:14.956501 7462 tablet_bootstrap.cc:654] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:14.985771 7459 ts_tablet_manager.cc:1434] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.286s user 0.008s sys 0.000s
I20260317 21:19:14.989954 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:14.990482 7329 raft_consensus.cc:3060] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:14.993634 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:14.995709 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:14.996450 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c9b7f7c864004b9abde0a0201e196165" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:14.996965 7184 raft_consensus.cc:3060] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:15.001814 7329 raft_consensus.cc:2468] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:15.004387 7261 leader_election.cc:343] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:15.003058 6971 leader_election.cc:343] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.009378 6973 leader_election.cc:304] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:15.010197 8384 raft_consensus.cc:2804] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:15.010795 8384 raft_consensus.cc:697] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:15.011659 8384 consensus_queue.cc:237] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:15.017378 6973 leader_election.cc:343] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: c9b7f7c864004b9abde0a0201e196165
I20260317 21:19:15.018388 8324 consensus_queue.cc:1048] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:19:15.020517 6971 consensus_peers.cc:597] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:14.961109 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 213257us
I20260317 21:19:15.025115 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 3e2f891e712341868d94bab1dd72a9e1 (table default.Table_0c31d9e5b0bd4454acccb920047a5e73 [id=8fbd9827525947d58d75c279e4b92b6a]) successfully deleted
I20260317 21:19:15.037690 8382 consensus_queue.cc:1048] T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:15.059104 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:15.061767 7459 tablet_bootstrap.cc:492] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:14.973186 8354 ts_tablet_manager.cc:1950] T 440cb8b112044a53804357056f1226f1 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:15.065470 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:15.075897 8372 raft_consensus.cc:493] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.076438 8372 raft_consensus.cc:515] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:15.087265 7119 leader_election.cc:343] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:15.087836 7119 leader_election.cc:304] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:15.088681 8364 raft_consensus.cc:2749] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:15.092482 7120 leader_election.cc:343] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:15.096860 7184 raft_consensus.cc:2468] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 2.
I20260317 21:19:15.098040 6971 leader_election.cc:304] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:15.099040 8340 raft_consensus.cc:2804] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:19:15.100037 8340 raft_consensus.cc:697] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:15.100852 8340 consensus_queue.cc:237] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.087347 8372 leader_election.cc:290] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:15.109580 6659 catalog_manager.cc:4969] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 7eb432b6bccf4e32a2fa9d4979586c52 because the tablet was not found. No further retry: Not found: Tablet not found: 7eb432b6bccf4e32a2fa9d4979586c52
I20260317 21:19:15.115511 8372 raft_consensus.cc:493] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.116019 8372 raft_consensus.cc:515] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.118312 8372 leader_election.cc:290] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.129951 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:15.143105 8359 tablet_replica.cc:333] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
W20260317 21:19:15.144997 6973 consensus_peers.cc:597] T 440cb8b112044a53804357056f1226f1 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 440cb8b112044a53804357056f1226f1. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:15.144968 8359 raft_consensus.cc:2243] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:15.145990 8359 raft_consensus.cc:2272] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:15.155493 7462 tablet_bootstrap.cc:492] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:15.155936 7462 ts_tablet_manager.cc:1403] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.486s user 0.008s sys 0.003s
W20260317 21:19:15.119033 7264 leader_election.cc:343] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.167608 7264 leader_election.cc:304] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:15.169386 7182 raft_consensus.cc:3060] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:15.176129 8292 catalog_manager.cc:1432] Term changed from 3 to 5 while waiting for master leader catchup. Not loading sys catalog metadata
I20260317 21:19:15.176837 8352 raft_consensus.cc:2749] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:15.117188 6661 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 440cb8b112044a53804357056f1226f1 (table default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]) successfully deleted
W20260317 21:19:15.181680 6661 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 64527.1us
I20260317 21:19:15.166641 7459 tablet_bootstrap.cc:654] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.208492 8292 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:19:15.217684 7459 tablet_bootstrap.cc:492] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:15.218113 7459 ts_tablet_manager.cc:1403] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.157s user 0.009s sys 0.001s
I20260317 21:19:15.222151 7459 raft_consensus.cc:359] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.222818 7459 raft_consensus.cc:385] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.223065 7459 raft_consensus.cc:740] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:15.228354 8395 log.cc:927] Time spent T 039d8bed256a43a6b98075b8430cbf3b P d3cb3b49e3e94537a6e434ea4e037c3f: Append to log took a long time: real 0.066s user 0.000s sys 0.001s
W20260317 21:19:15.168958 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 49960.7us
I20260317 21:19:15.228704 7459 consensus_queue.cc:260] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:15.229524 7264 leader_election.cc:343] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.229935 7264 leader_election.cc:304] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:15.230597 8352 raft_consensus.cc:2749] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:15.234287 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0fd3e35e25bf444cbd821ebd9f5ee018" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:15.234560 7459 ts_tablet_manager.cc:1434] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.016s user 0.005s sys 0.002s
I20260317 21:19:15.239154 7459 tablet_bootstrap.cc:492] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:15.244798 7459 tablet_bootstrap.cc:654] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.259128 7459 tablet_bootstrap.cc:492] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:15.259553 7459 ts_tablet_manager.cc:1403] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.004s
I20260317 21:19:15.261821 7459 raft_consensus.cc:359] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.262458 7459 raft_consensus.cc:385] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.262713 7459 raft_consensus.cc:740] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.263375 7459 consensus_queue.cc:260] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.265581 7459 ts_tablet_manager.cc:1434] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.006s user 0.006s sys 0.001s
I20260317 21:19:15.192426 7462 raft_consensus.cc:359] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.266325 7462 raft_consensus.cc:385] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.266593 7462 raft_consensus.cc:740] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.267174 7462 consensus_queue.cc:260] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.269254 7462 ts_tablet_manager.cc:1434] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.113s user 0.007s sys 0.000s
I20260317 21:19:15.270112 7462 tablet_bootstrap.cc:492] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:15.270720 7119 leader_election.cc:343] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:15.271308 7119 leader_election.cc:304] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:15.283025 7467 tablet_bootstrap.cc:492] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:15.283598 7467 ts_tablet_manager.cc:1403] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.350s user 0.012s sys 0.006s
W20260317 21:19:15.280802 7119 raft_consensus.cc:2686] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:19:15.287976 7119 leader_election.cc:304] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:15.290613 8372 raft_consensus.cc:2749] T 0fd3e35e25bf444cbd821ebd9f5ee018 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:15.294010 6660 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 8a44a297079347bab6f8be6abe332bfa (table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8a44a297079347bab6f8be6abe332bfa consensus state has no leader
I20260317 21:19:15.303449 8340 raft_consensus.cc:493] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.304242 8340 raft_consensus.cc:515] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.306764 8340 leader_election.cc:290] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.307837 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:15.307916 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:15.308411 7328 raft_consensus.cc:2468] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:15.309630 6973 leader_election.cc:304] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:15.310539 6971 leader_election.cc:343] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.311734 8340 raft_consensus.cc:2804] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:15.312151 8340 raft_consensus.cc:493] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:15.312525 8340 raft_consensus.cc:3060] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:15.322209 8340 raft_consensus.cc:515] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.326851 7467 raft_consensus.cc:359] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.327541 7467 raft_consensus.cc:385] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.327797 7467 raft_consensus.cc:740] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.338323 8340 leader_election.cc:290] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.339704 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:15.340198 7328 raft_consensus.cc:3060] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:15.347947 7328 raft_consensus.cc:2468] T e06a1488c60f4d1387417ce30579fbf4 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:15.349325 6973 leader_election.cc:304] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:15.350096 8340 raft_consensus.cc:2804] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:15.350839 8340 raft_consensus.cc:697] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:15.351647 8340 consensus_queue.cc:237] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.372442 7462 tablet_bootstrap.cc:654] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.378297 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:15.336621 8359 ts_tablet_manager.cc:1916] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:15.391661 7459 tablet_bootstrap.cc:492] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:15.392244 6971 leader_election.cc:343] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.328357 7467 consensus_queue.cc:260] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.403045 7459 tablet_bootstrap.cc:654] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.420501 7462 tablet_bootstrap.cc:492] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:15.421197 7462 ts_tablet_manager.cc:1403] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.151s user 0.014s sys 0.000s
I20260317 21:19:15.423913 7462 raft_consensus.cc:359] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.396010 7467 ts_tablet_manager.cc:1434] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.112s user 0.005s sys 0.000s
I20260317 21:19:15.428143 7459 tablet_bootstrap.cc:492] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:15.428543 7459 ts_tablet_manager.cc:1403] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.003s
I20260317 21:19:15.433414 7467 tablet_bootstrap.cc:492] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:15.424588 7462 raft_consensus.cc:385] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.447149 7462 raft_consensus.cc:740] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.438980 7459 raft_consensus.cc:359] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.448520 7459 raft_consensus.cc:385] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.448948 7459 raft_consensus.cc:740] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.458767 7462 consensus_queue.cc:260] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.460197 7459 consensus_queue.cc:260] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.467578 8079 mvcc.cc:204] Tried to move back new op lower bound from 7265412483962437632 to 7265412415029694464. Current Snapshot: MvccSnapshot[applied={T|T < 7265412483962437632 or (T in {7265412483962437632})}]
I20260317 21:19:15.482950 8410 tablet.cc:1723] T 221c71b908f942ca97f111f857f0335a P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:15.484666 8411 tablet.cc:1723] T 221c71b908f942ca97f111f857f0335a P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:15.501932 7467 tablet_bootstrap.cc:654] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.501958 8412 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:15.503396 7462 ts_tablet_manager.cc:1434] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.082s user 0.006s sys 0.000s
I20260317 21:19:15.503873 8412 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:15.517792 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:15.515818 8412 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
W20260317 21:19:15.537765 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 75721.2us
I20260317 21:19:15.542316 8337 raft_consensus.cc:493] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.542804 8337 raft_consensus.cc:515] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.546838 8383 raft_consensus.cc:493] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.547323 8383 raft_consensus.cc:515] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.551766 7184 raft_consensus.cc:3060] T f9e98c10a83e429a98a0fb4328e1b0ee P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:15.558357 8383 leader_election.cc:290] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.559360 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18e4b69574e44eb28a735edc64e97542" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:15.559947 7329 raft_consensus.cc:2468] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:15.561021 6973 leader_election.cc:304] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:15.561826 8383 raft_consensus.cc:2804] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:15.562620 8383 raft_consensus.cc:493] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:15.563037 8383 raft_consensus.cc:3060] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:15.569077 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:15.570750 7918 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:58158 (request call id 5) took 1271 ms. Trace:
I20260317 21:19:15.573009 8337 leader_election.cc:290] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.574586 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:15.575776 6973 leader_election.cc:343] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.575780 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18e4b69574e44eb28a735edc64e97542" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:15.577726 7892 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.65:54487 (request call id 55) took 914 ms (client timeout 1000 ms). Trace:
W20260317 21:19:15.577826 6971 leader_election.cc:343] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.578207 6971 leader_election.cc:304] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:15.579210 6971 leader_election.cc:343] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.580068 8355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:15.579052 8366 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:15.580777 8366 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:15.580588 8355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
W20260317 21:19:15.581960 7890 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.66:36101 (request call id 56) took 1200 ms (client timeout 1000 ms). Trace:
I20260317 21:19:15.582204 7459 ts_tablet_manager.cc:1434] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.153s user 0.006s sys 0.000s
W20260317 21:19:15.582260 7890 rpcz_store.cc:269] 0317 21:19:14.381848 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.382022 (+ 174us) service_pool.cc:225] Handling call
0317 21:19:15.570773 (+1188751us) spinlock_profiling.cc:243] Waited 1.19 s on lock 0x7b5400030554. stack: 00007f6a5dec8423 00007f6a5dec8320 00007f6a5da3d574 0000000000509544 00000000005094d9 00007f6a64f242c0 00007f6a64fc2e6a 00007f6a66cc521c 00007f6a66dbdb2e 00007f6a649d94a4 00007f6a649d9432 00007f6a649d9361 00007f6a649d92dc 00007f6a649d8592 00007f6a62cb634c 00007f6a62cb5786
0317 21:19:15.581784 (+ 11011us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2729361280}
I20260317 21:19:15.582988 8355 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
W20260317 21:19:15.580149 7892 rpcz_store.cc:269] 0317 21:19:14.663531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.663683 (+ 152us) service_pool.cc:225] Handling call
0317 21:19:15.574984 (+911301us) spinlock_profiling.cc:243] Waited 909 ms on lock 0x7b5400030554. stack: 00007f6a5dec8423 00007f6a5dec8320 00007f6a5da3d574 0000000000509544 00000000005094d9 00007f6a64f242c0 00007f6a64fc2e6a 00007f6a66cc521c 00007f6a66dbdb2e 00007f6a649d94a4 00007f6a649d9432 00007f6a649d9361 00007f6a649d92dc 00007f6a649d8592 00007f6a62cb634c 00007f6a62cb5786
0317 21:19:15.577628 (+ 2644us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2089583488}
I20260317 21:19:15.585068 8383 raft_consensus.cc:515] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.587795 7895 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:58164 (request call id 27) took 1276 ms. Trace:
I20260317 21:19:15.588101 7895 rpcz_store.cc:276] 0317 21:19:14.311433 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.315527 (+ 4094us) service_pool.cc:225] Handling call
0317 21:19:15.587015 (+1271488us) spinlock_profiling.cc:243] Waited 1.27 s on lock 0x7b5400030554. stack: 00007f6a5dec8423 00007f6a5dec8320 00007f6a5da3d574 0000000000509544 00000000005094d9 00007f6a64f242c0 00007f6a64fc2e6a 00007f6a66cc521c 00007f6a66dc0de9 00007f6a649dba24 00007f6a649db9b2 00007f6a649db8e1 00007f6a649db85c 00007f6a649dab12 00007f6a62cb634c 00007f6a62cb5786
0317 21:19:15.587737 (+ 722us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2920785152}
I20260317 21:19:15.590219 7462 tablet_bootstrap.cc:492] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:15.590131 7888 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.6.122.67:56885 (request call id 34) took 955 ms (client timeout 1000 ms). Trace:
W20260317 21:19:15.590468 7888 rpcz_store.cc:269] 0317 21:19:14.634685 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.636267 (+ 1582us) service_pool.cc:225] Handling call
0317 21:19:15.584437 (+948170us) spinlock_profiling.cc:243] Waited 945 ms on lock 0x7b5400030554. stack: 00007f6a5dec8423 00007f6a5dec8320 00007f6a5da3d574 0000000000509544 00000000005094d9 00007f6a64f242c0 00007f6a64fc2e6a 00007f6a66cc521c 00007f6a66dbdb2e 00007f6a649d94a4 00007f6a649d9432 00007f6a649d9361 00007f6a649d92dc 00007f6a649d8592 00007f6a62cb634c 00007f6a62cb5786
0317 21:19:15.590070 (+ 5633us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2173976832}
I20260317 21:19:15.593087 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:15.593756 6683 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e1c983d0224428a9a7b147a0e5e4ef in term 5.
I20260317 21:19:15.598212 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:19:15.606165 8355 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Leader pre-election won for term 6
I20260317 21:19:15.606599 8355 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Starting leader election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:15.607023 8355 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 5 FOLLOWER]: Advancing to term 6
I20260317 21:19:15.598989 8324 raft_consensus.cc:2749] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:15.618733 8383 leader_election.cc:290] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.603114 7459 tablet_bootstrap.cc:492] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:15.619827 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18e4b69574e44eb28a735edc64e97542" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:15.620391 7329 raft_consensus.cc:3060] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:15.630250 7329 raft_consensus.cc:2468] T 18e4b69574e44eb28a735edc64e97542 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:15.631313 6973 leader_election.cc:304] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:15.632047 7462 tablet_bootstrap.cc:654] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.635339 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18e4b69574e44eb28a735edc64e97542" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:15.636623 6971 leader_election.cc:343] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.643452 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9]
I20260317 21:19:15.647513 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e]
I20260317 21:19:15.653223 8383 raft_consensus.cc:2804] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:15.654033 8383 raft_consensus.cc:697] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:15.655108 8383 consensus_queue.cc:237] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.640285 8355 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:15.669463 7459 tablet_bootstrap.cc:654] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.678146 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:19:15.693163 8415 tablet.cc:1723] T 221c71b908f942ca97f111f857f0335a P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:15.697970 8355 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 6 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:15.700135 7462 tablet_bootstrap.cc:492] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:15.700565 7462 ts_tablet_manager.cc:1403] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.111s user 0.006s sys 0.003s
I20260317 21:19:15.703847 7462 raft_consensus.cc:359] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.717088 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1342223726fb480aba4932d06e045aaf [id=06e60ad198054f26b5512c8f53e70753]
I20260317 21:19:15.718847 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017]
I20260317 21:19:15.719339 7467 tablet_bootstrap.cc:492] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:15.719790 7467 ts_tablet_manager.cc:1403] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.287s user 0.009s sys 0.002s
I20260317 21:19:15.722904 7467 raft_consensus.cc:359] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.723577 7467 raft_consensus.cc:385] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.723830 7467 raft_consensus.cc:740] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.710042 7459 tablet_bootstrap.cc:492] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:15.728073 7459 ts_tablet_manager.cc:1403] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.125s user 0.012s sys 0.000s
I20260317 21:19:15.730445 7459 raft_consensus.cc:359] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.731192 7459 raft_consensus.cc:385] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.731492 7459 raft_consensus.cc:740] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.732501 7459 consensus_queue.cc:260] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.738164 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]
I20260317 21:19:15.704502 7462 raft_consensus.cc:385] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.742755 7462 raft_consensus.cc:740] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.743783 7462 consensus_queue.cc:260] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.746860 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7]
I20260317 21:19:15.759584 7459 ts_tablet_manager.cc:1434] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.031s user 0.003s sys 0.001s
W20260317 21:19:15.779093 7261 consensus_peers.cc:597] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:15.712411 8359 ts_tablet_manager.cc:1929] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:15.791977 8359 log.cc:1199] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/5314c562bc9d457a8e1f4930e8d3a7ff
I20260317 21:19:15.792737 8359 ts_tablet_manager.cc:1950] T 5314c562bc9d457a8e1f4930e8d3a7ff P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:15.801571 7467 consensus_queue.cc:260] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.806274 7462 ts_tablet_manager.cc:1434] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.105s user 0.007s sys 0.000s
I20260317 21:19:15.813194 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_15292a1c98be431b9d3cacbc7e767c75 [id=0f32cb4e70be48c2a548e9e2d4805822]
I20260317 21:19:15.822232 7459 tablet_bootstrap.cc:492] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:15.822867 7462 tablet_bootstrap.cc:492] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:15.838660 7459 tablet_bootstrap.cc:654] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.846060 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]
I20260317 21:19:15.847702 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]
I20260317 21:19:15.849515 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]
I20260317 21:19:15.852582 7467 ts_tablet_manager.cc:1434] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.132s user 0.002s sys 0.005s
I20260317 21:19:15.855551 7459 tablet_bootstrap.cc:492] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:15.855963 7459 ts_tablet_manager.cc:1403] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.004s
I20260317 21:19:15.878118 7462 tablet_bootstrap.cc:654] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.893122 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193 [id=121b3b215a0b4914a3f18167636ce46e]
I20260317 21:19:15.891855 7459 raft_consensus.cc:359] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.894593 7459 raft_consensus.cc:385] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:15.894874 7459 raft_consensus.cc:740] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.895546 7459 consensus_queue.cc:260] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.898372 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373]
I20260317 21:19:15.902117 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9]
I20260317 21:19:15.902899 8359 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 326) took 1481 ms. Trace:
I20260317 21:19:15.903309 8359 rpcz_store.cc:276] 0317 21:19:14.421220 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.421997 (+ 777us) service_pool.cc:225] Handling call
0317 21:19:15.109274 (+687277us) ts_tablet_manager.cc:1151] Deleting tablet 5314c562bc9d457a8e1f4930e8d3a7ff
0317 21:19:15.142742 (+ 33468us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:15.479927 (+337185us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:15.712160 (+232233us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:15.902738 (+190578us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":686606}
I20260317 21:19:15.905069 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 5314c562bc9d457a8e1f4930e8d3a7ff (table default.Table_aef4da11f840465eb7fc80e6423d7454 [id=91eadfdf7d094f9fa83003dace842023]) successfully deleted
I20260317 21:19:15.906064 8359 tablet_replica.cc:333] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:15.906950 8359 raft_consensus.cc:2243] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:15.907943 8359 pending_rounds.cc:70] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Trying to abort 1 pending ops.
I20260317 21:19:15.909305 8359 pending_rounds.cc:77] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412512525860864 op_type: NO_OP noop_request { }
I20260317 21:19:15.908990 7467 tablet_bootstrap.cc:492] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:15.910145 8359 raft_consensus.cc:2889] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:15.923728 8359 raft_consensus.cc:2272] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:15.911767 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05]
I20260317 21:19:15.935135 8372 raft_consensus.cc:493] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.935604 8372 raft_consensus.cc:515] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.936096 7462 tablet_bootstrap.cc:492] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:15.936473 7462 ts_tablet_manager.cc:1403] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.001s
I20260317 21:19:15.934227 8383 raft_consensus.cc:493] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:15.940920 8372 leader_election.cc:290] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.938845 7462 raft_consensus.cc:359] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.941205 8383 raft_consensus.cc:515] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.941915 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:15.942965 8383 leader_election.cc:290] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:15.944367 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:15.945741 7462 raft_consensus.cc:385] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 21:19:15.945832 6971 leader_election.cc:343] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:15.928552 7459 ts_tablet_manager.cc:1434] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.072s user 0.006s sys 0.001s
I20260317 21:19:15.946101 7462 raft_consensus.cc:740] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:15.946764 7462 consensus_queue.cc:260] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:15.949456 7459 tablet_bootstrap.cc:492] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:15.960311 7462 ts_tablet_manager.cc:1434] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.024s user 0.004s sys 0.000s
I20260317 21:19:15.960939 7184 raft_consensus.cc:1275] T c9b7f7c864004b9abde0a0201e196165 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Refusing update from remote peer d3cb3b49e3e94537a6e434ea4e037c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 21:19:15.961498 7462 tablet_bootstrap.cc:492] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:15.963562 8337 consensus_queue.cc:1048] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:15.973621 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06]
I20260317 21:19:15.976336 7467 tablet_bootstrap.cc:654] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.976811 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19]
I20260317 21:19:15.980762 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f]
I20260317 21:19:15.986742 7459 tablet_bootstrap.cc:654] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.001163 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7]
I20260317 21:19:16.003057 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]
I20260317 21:19:16.006193 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]
I20260317 21:19:16.015496 7459 tablet_bootstrap.cc:492] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:16.000152 8359 ts_tablet_manager.cc:1916] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:16.031046 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_15fe620085e9499295d76e3a7b3df064 [id=242dcdafc2b74e1ca17839f96e45af88]
I20260317 21:19:16.034365 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_c928476e679c41589639228020653020 [id=25a841c1c6e2466eb130c7fac5dd1e68]
I20260317 21:19:16.039930 8324 raft_consensus.cc:493] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.047016 8324 raft_consensus.cc:515] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.047926 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5ebd4f0e4c284b4caac2e5f0cddf4e07 [id=2d22abe029d348c484e6aff5df8b3d29]
I20260317 21:19:16.050935 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:16.049082 8324 leader_election.cc:290] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:16.050935 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "99c6483618044272890b59f6b7447081" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:16.058070 7326 raft_consensus.cc:2393] T 99c6483618044272890b59f6b7447081 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:16.059041 7462 tablet_bootstrap.cc:654] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.061333 7119 leader_election.cc:304] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:16.062984 6973 consensus_peers.cc:597] T c9b7f7c864004b9abde0a0201e196165 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c9b7f7c864004b9abde0a0201e196165. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:16.066044 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:16.052353 7328 raft_consensus.cc:2468] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:16.067050 7329 raft_consensus.cc:2468] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:16.069427 6973 leader_election.cc:304] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:16.070448 6973 leader_election.cc:304] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:16.071162 8337 raft_consensus.cc:2804] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.071480 8337 raft_consensus.cc:493] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:16.071790 8337 raft_consensus.cc:3060] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.076128 8384 raft_consensus.cc:2804] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.076576 8384 raft_consensus.cc:493] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:16.076956 8384 raft_consensus.cc:3060] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.097347 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:16.099061 6971 leader_election.cc:343] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.100314 7459 ts_tablet_manager.cc:1403] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.151s user 0.010s sys 0.001s
W20260317 21:19:16.100967 6661 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.589s (SENT) [suppressed 2 similar messages]
W20260317 21:19:16.105531 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 44270.1us
I20260317 21:19:16.125205 8337 raft_consensus.cc:515] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.128151 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:16.129709 6971 leader_election.cc:343] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.130865 8384 raft_consensus.cc:515] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.133764 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:16.135864 6971 leader_election.cc:343] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.136916 8384 leader_election.cc:290] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:16.142903 7459 raft_consensus.cc:359] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.145196 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:16.145766 7327 raft_consensus.cc:3060] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.147823 7184 raft_consensus.cc:3060] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.154443 7462 tablet_bootstrap.cc:492] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:16.154939 7462 ts_tablet_manager.cc:1403] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.194s user 0.010s sys 0.007s
I20260317 21:19:16.156342 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:16.156903 7328 raft_consensus.cc:3060] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.156968 7459 raft_consensus.cc:385] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:16.157270 7459 raft_consensus.cc:740] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.158039 7459 consensus_queue.cc:260] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.154214 8337 leader_election.cc:290] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:16.164479 7328 raft_consensus.cc:2468] T 203f33157a0d4b82aab44da452aa46dd P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:16.167133 7327 raft_consensus.cc:2468] T e586f11c37184506a15d726b4eb333be P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:16.169353 8352 consensus_queue.cc:1048] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:16.173439 8430 raft_consensus.cc:493] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.174186 8430 raft_consensus.cc:515] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.178862 6973 leader_election.cc:304] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:16.180851 8337 raft_consensus.cc:2804] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:16.185766 8352 raft_consensus.cc:493] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.186513 8352 raft_consensus.cc:515] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.187299 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d94620882364ed3929ee314814910c9" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:16.188050 7327 raft_consensus.cc:2468] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:16.188776 8352 leader_election.cc:290] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:16.190646 6973 leader_election.cc:304] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:16.191691 8435 raft_consensus.cc:2804] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:16.192281 8435 raft_consensus.cc:697] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:16.192219 7119 leader_election.cc:304] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:16.193209 8337 raft_consensus.cc:697] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:16.194547 8337 consensus_queue.cc:237] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.193192 8435 consensus_queue.cc:237] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.161093 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b]
I20260317 21:19:16.176697 8430 leader_election.cc:290] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:16.203712 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc]
I20260317 21:19:16.207438 7467 tablet_bootstrap.cc:492] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:16.207875 7467 ts_tablet_manager.cc:1403] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.299s user 0.004s sys 0.009s
I20260317 21:19:16.205302 8430 raft_consensus.cc:2804] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.213490 8430 raft_consensus.cc:493] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:16.215157 8430 raft_consensus.cc:3060] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.218250 8352 raft_consensus.cc:493] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.218748 8352 raft_consensus.cc:515] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.220831 8352 leader_election.cc:290] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:16.204596 7459 ts_tablet_manager.cc:1434] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.081s user 0.005s sys 0.001s
I20260317 21:19:16.227111 7459 tablet_bootstrap.cc:492] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:16.228430 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "740ae685629041f39bbbc8ce715d662e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:16.228729 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d94620882364ed3929ee314814910c9" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:16.220280 7462 raft_consensus.cc:359] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.242304 7462 raft_consensus.cc:385] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:16.242617 7462 raft_consensus.cc:740] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.249461 7462 consensus_queue.cc:260] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.250306 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_db13c9846b0049dca4c784a97c434893 [id=305ca4adf2674d4c8cfa2b63e2316871]
I20260317 21:19:16.252063 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0]
I20260317 21:19:16.262447 7467 raft_consensus.cc:359] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.263182 7467 raft_consensus.cc:385] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 21:19:16.263984 6971 consensus_peers.cc:597] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:16.263441 7467 raft_consensus.cc:740] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.266156 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]
I20260317 21:19:16.266094 7467 consensus_queue.cc:260] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:16.283780 7821 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.585s (SENT)
I20260317 21:19:16.288160 7462 ts_tablet_manager.cc:1434] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.133s user 0.008s sys 0.001s
I20260317 21:19:16.292312 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]
I20260317 21:19:16.290508 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "740ae685629041f39bbbc8ce715d662e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:16.296633 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479]
I20260317 21:19:16.297271 7467 ts_tablet_manager.cc:1434] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.089s user 0.004s sys 0.004s
I20260317 21:19:16.299209 7467 tablet_bootstrap.cc:492] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:16.296713 7182 raft_consensus.cc:2468] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:16.301095 7462 tablet_bootstrap.cc:492] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:16.302070 7034 raft_consensus.cc:2468] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:16.302915 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4cee7952cf5428b98340789a48d2da2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:16.303450 7038 raft_consensus.cc:2468] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
W20260317 21:19:16.305124 7264 leader_election.cc:343] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.307476 7264 leader_election.cc:304] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:16.308148 8352 raft_consensus.cc:2804] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.308470 8352 raft_consensus.cc:493] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:16.308804 8352 raft_consensus.cc:3060] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.312242 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67]
I20260317 21:19:16.313906 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_54698fcc6a7a46deaba83055d4a1c78d [id=3e906ea0e8af47e383d0403c155be644]
I20260317 21:19:16.316478 8337 consensus_queue.cc:1048] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:16.327359 8446 log.cc:927] Time spent T 252f728d183f440a9ef42894508d5755 P d3cb3b49e3e94537a6e434ea4e037c3f: Append to log took a long time: real 0.089s user 0.001s sys 0.000s
I20260317 21:19:16.334029 8450 tablet.cc:1723] T 63a27911e0e545f5811eb075997d0b0f P 2f539d1a4a584b8ca83928be49369bf4: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:16.336123 8438 raft_consensus.cc:493] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.336714 8438 raft_consensus.cc:515] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.339087 8438 leader_election.cc:290] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:16.306501 8411 tablet.cc:1723] T 63a27911e0e545f5811eb075997d0b0f P c0ef92fef382401a9564c17e0c223f6a: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:16.326370 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4cee7952cf5428b98340789a48d2da2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:16.329130 8439 raft_consensus.cc:493] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.349581 8439 raft_consensus.cc:515] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.351717 8439 leader_election.cc:290] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:16.340812 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:16.355041 7119 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:16.332353 8449 tablet.cc:1723] T 63a27911e0e545f5811eb075997d0b0f P d3cb3b49e3e94537a6e434ea4e037c3f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20260317 21:19:16.350777 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:16.364100 7261 leader_election.cc:343] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:16.365072 7261 leader_election.cc:343] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.372408 8431 raft_consensus.cc:2749] T 99c6483618044272890b59f6b7447081 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:16.374315 7264 leader_election.cc:343] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.374761 7264 leader_election.cc:304] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:16.375988 8383 consensus_queue.cc:1048] T 99c6483618044272890b59f6b7447081 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:16.380878 7459 tablet_bootstrap.cc:654] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.344110 8352 raft_consensus.cc:515] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.398097 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4cee7952cf5428b98340789a48d2da2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:16.398599 7036 raft_consensus.cc:3060] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.398515 7261 leader_election.cc:304] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:16.400036 8359 ts_tablet_manager.cc:1929] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:16.406564 7036 raft_consensus.cc:2468] T a4cee7952cf5428b98340789a48d2da2 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:16.406816 8359 log.cc:1199] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/bcd99ca9f6c540e48e29d8659fbfe777
I20260317 21:19:16.409109 7264 leader_election.cc:304] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:16.387137 8451 raft_consensus.cc:2749] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:16.422050 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4cee7952cf5428b98340789a48d2da2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:16.426653 8453 raft_consensus.cc:2804] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
W20260317 21:19:16.427345 7261 leader_election.cc:343] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.427843 8453 raft_consensus.cc:697] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:16.434424 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]
I20260317 21:19:16.443392 8431 raft_consensus.cc:493] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.443918 8431 raft_consensus.cc:515] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.445971 8431 leader_election.cc:290] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:16.453869 6971 consensus_peers.cc:597] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:16.455691 7459 tablet_bootstrap.cc:492] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:16.456248 7459 ts_tablet_manager.cc:1403] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.229s user 0.007s sys 0.006s
I20260317 21:19:16.458477 7467 tablet_bootstrap.cc:654] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.461702 7459 raft_consensus.cc:359] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.396965 8352 leader_election.cc:290] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:16.489319 7120 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:16.482095 8454 raft_consensus.cc:2804] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.490934 8454 raft_consensus.cc:493] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:16.492344 8454 raft_consensus.cc:3060] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.487656 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_6157fa5cda0e4c34817e01c6bdde4904 [id=4a69de09e4be4630994811c7e432c315]
W20260317 21:19:16.493278 7120 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:16.493682 7120 leader_election.cc:304] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:16.494379 8431 raft_consensus.cc:2749] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:16.501243 8453 consensus_queue.cc:237] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.407586 8359 ts_tablet_manager.cc:1950] T bcd99ca9f6c540e48e29d8659fbfe777 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:16.510238 7462 tablet_bootstrap.cc:654] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.476128 8430 raft_consensus.cc:515] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.521150 7459 raft_consensus.cc:385] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:16.521522 7459 raft_consensus.cc:740] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.522509 7459 consensus_queue.cc:260] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.524942 7459 ts_tablet_manager.cc:1434] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.068s user 0.005s sys 0.000s
I20260317 21:19:16.518939 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]
I20260317 21:19:16.525923 7459 tablet_bootstrap.cc:492] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:16.534332 8359 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 327) took 2112 ms. Trace:
I20260317 21:19:16.534749 8359 rpcz_store.cc:276] 0317 21:19:14.422910 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.423472 (+ 562us) service_pool.cc:225] Handling call
0317 21:19:15.905892 (+1482420us) ts_tablet_manager.cc:1151] Deleting tablet bcd99ca9f6c540e48e29d8659fbfe777
0317 21:19:15.905941 (+ 49us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:16.270795 (+364854us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:16.399785 (+128990us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:16.534253 (+134468us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1481879}
I20260317 21:19:16.536757 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet bcd99ca9f6c540e48e29d8659fbfe777 (table default.Table_fcec6e585d6245ff828c8c0c98450d63 [id=d238f6cb466c49d6af5ae380e967b054]) successfully deleted
I20260317 21:19:16.543027 8454 raft_consensus.cc:515] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:16.549062 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 140019us
I20260317 21:19:16.547904 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "740ae685629041f39bbbc8ce715d662e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:16.550580 8456 raft_consensus.cc:493] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.555037 8430 leader_election.cc:290] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:16.555882 8456 raft_consensus.cc:515] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.559562 8456 leader_election.cc:290] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:16.562582 7119 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:16.563042 7119 leader_election.cc:304] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:16.564605 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d94620882364ed3929ee314814910c9" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:16.565227 7327 raft_consensus.cc:3060] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.568876 8430 raft_consensus.cc:2749] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:16.574047 8454 leader_election.cc:290] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:16.594893 7459 tablet_bootstrap.cc:654] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.610831 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8d94620882364ed3929ee314814910c9" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:16.611403 7038 raft_consensus.cc:3060] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.619894 7038 raft_consensus.cc:2468] T 8d94620882364ed3929ee314814910c9 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:16.627045 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "740ae685629041f39bbbc8ce715d662e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:16.628708 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:16.629279 7036 raft_consensus.cc:2468] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:16.622277 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:16.638598 8337 consensus_queue.cc:1048] T 18e4b69574e44eb28a735edc64e97542 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:16.640223 7261 leader_election.cc:343] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.622324 8359 tablet_replica.cc:333] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:16.647751 7120 leader_election.cc:304] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:16.648516 8430 raft_consensus.cc:2804] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:16.649029 8430 raft_consensus.cc:697] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:16.650017 8430 consensus_queue.cc:237] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.665005 7459 tablet_bootstrap.cc:492] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:16.665529 7459 ts_tablet_manager.cc:1403] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.140s user 0.010s sys 0.001s
I20260317 21:19:16.671696 8359 raft_consensus.cc:2243] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:16.667906 7459 raft_consensus.cc:359] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.680665 7459 raft_consensus.cc:385] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:16.682026 7459 raft_consensus.cc:740] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.688339 8359 raft_consensus.cc:2272] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:16.690474 8359 ts_tablet_manager.cc:1916] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:19:16.690997 7264 leader_election.cc:343] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.691581 7264 leader_election.cc:304] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:16.609362 7184 raft_consensus.cc:3060] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.695329 7327 raft_consensus.cc:2468] T 8d94620882364ed3929ee314814910c9 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:16.636106 7467 tablet_bootstrap.cc:492] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:16.687335 7459 consensus_queue.cc:260] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.697134 7467 ts_tablet_manager.cc:1403] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.398s user 0.010s sys 0.003s
I20260317 21:19:16.699106 7459 ts_tablet_manager.cc:1434] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.033s user 0.005s sys 0.000s
I20260317 21:19:16.699501 7467 raft_consensus.cc:359] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.700139 7467 raft_consensus.cc:385] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:16.700397 7467 raft_consensus.cc:740] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.701112 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]
I20260317 21:19:16.702775 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2]
I20260317 21:19:16.704319 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_61b756f1959b46f393c2a726704e78aa [id=520224705f0d45feb82bb3a96db2cca0]
I20260317 21:19:16.696462 8454 raft_consensus.cc:2804] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.708901 8337 consensus_queue.cc:1048] T e06a1488c60f4d1387417ce30579fbf4 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:16.712986 8454 raft_consensus.cc:493] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:16.713356 8454 raft_consensus.cc:3060] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:16.720991 6661 leader_election.cc:336] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779): Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.589s (SENT)
I20260317 21:19:16.721463 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:16.726958 8454 raft_consensus.cc:515] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.729183 7459 tablet_bootstrap.cc:492] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:16.735212 7462 tablet_bootstrap.cc:492] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:16.747046 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_525a8435bbe14a13a70b022bb2c07f59 [id=539b531cf22a4816b0aff0f3908b0439]
I20260317 21:19:16.748900 8466 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 21:19:16.735754 7462 ts_tablet_manager.cc:1403] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.447s user 0.011s sys 0.004s
I20260317 21:19:16.756138 8454 leader_election.cc:290] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:16.770777 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:16.771340 7036 raft_consensus.cc:3060] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.744360 7184 raft_consensus.cc:2468] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:16.780447 7036 raft_consensus.cc:2468] T 2a221dfad830428db2329d6e543f41dc P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:16.789021 7467 consensus_queue.cc:260] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.799263 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:16.799784 6660 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 4495961a559549629dedf8ce032abc9b (table default.Table_525a8435bbe14a13a70b022bb2c07f59 [id=539b531cf22a4816b0aff0f3908b0439], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4495961a559549629dedf8ce032abc9b consensus state has no leader
I20260317 21:19:16.800489 7467 ts_tablet_manager.cc:1434] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.103s user 0.004s sys 0.004s
I20260317 21:19:16.801522 7467 tablet_bootstrap.cc:492] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:16.801575 7179 raft_consensus.cc:3060] T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:16.802906 7264 leader_election.cc:304] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:16.803743 8454 raft_consensus.cc:2804] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:16.804709 8454 raft_consensus.cc:697] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:16.809713 8454 consensus_queue.cc:237] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:16.820937 7261 leader_election.cc:343] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.760095 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 [id=54eed51395b74f46b69a56cfa73f3724]
I20260317 21:19:16.852317 7261 leader_election.cc:304] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters: d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:16.854149 7459 tablet_bootstrap.cc:700] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent opening tablet: real 0.111s user 0.002s sys 0.001s
I20260317 21:19:16.860316 8456 raft_consensus.cc:2804] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:16.860841 8456 raft_consensus.cc:697] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:16.865401 7462 raft_consensus.cc:359] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.865659 8456 consensus_queue.cc:237] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.874778 7459 tablet_bootstrap.cc:654] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.866098 7462 raft_consensus.cc:385] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:16.883821 7462 raft_consensus.cc:740] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:16.886256 7462 consensus_queue.cc:260] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.889181 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072]
I20260317 21:19:16.891460 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]
I20260317 21:19:16.915170 8456 raft_consensus.cc:493] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:16.915433 7467 tablet_bootstrap.cc:654] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:16.915643 8456 raft_consensus.cc:515] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:16.917963 7459 tablet_bootstrap.cc:492] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:16.918396 7459 ts_tablet_manager.cc:1403] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.204s user 0.008s sys 0.004s
I20260317 21:19:16.918740 7462 ts_tablet_manager.cc:1434] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.168s user 0.006s sys 0.001s
I20260317 21:19:16.922260 7462 tablet_bootstrap.cc:492] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:16.923046 8456 leader_election.cc:290] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:16.923002 7821 leader_election.cc:336] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 6 election: RPC error from VoteRequest() call to peer 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779): Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.585s (SENT)
I20260317 21:19:16.923477 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 6 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:16.923776 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2dccb915fad4666bd7978f8c416db44" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:16.924515 7036 raft_consensus.cc:2468] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:16.925758 7264 leader_election.cc:304] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:16.927747 8456 raft_consensus.cc:2804] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:16.928325 8456 raft_consensus.cc:493] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260317 21:19:16.929626 8470 log.cc:927] Time spent T 440cb8b112044a53804357056f1226f1 P c0ef92fef382401a9564c17e0c223f6a: Append to log took a long time: real 0.079s user 0.001s sys 0.000s
I20260317 21:19:16.932746 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2dccb915fad4666bd7978f8c416db44" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:16.934013 8456 raft_consensus.cc:3060] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:16.936739 7261 leader_election.cc:343] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:16.937386 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6]
I20260317 21:19:16.948627 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_4827867befa741f69f6ab15938cfe436 [id=5b18e88ee5b64c1fb5f7e361c866f44a]
I20260317 21:19:16.950489 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]
I20260317 21:19:16.963178 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547]
I20260317 21:19:16.964959 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793]
I20260317 21:19:16.966639 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611]
I20260317 21:19:16.968845 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e]
I20260317 21:19:16.971462 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]
I20260317 21:19:16.974649 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93]
I20260317 21:19:16.978561 8359 ts_tablet_manager.cc:1929] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:16.979197 8359 log.cc:1199] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/5b066a6a5c864beb8da59bd97bacd309
I20260317 21:19:17.010658 8473 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Leader election lost for term 6. Reason: could not achieve majority
W20260317 21:19:17.037076 7821 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 114128us
I20260317 21:19:17.045428 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]
I20260317 21:19:17.047065 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]
W20260317 21:19:17.050709 6971 consensus_peers.cc:597] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:17.018107 7459 raft_consensus.cc:359] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.035940 7462 tablet_bootstrap.cc:654] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:17.054667 8359 ts_tablet_manager.cc:1950] T 5b066a6a5c864beb8da59bd97bacd309 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:17.057272 7467 tablet_bootstrap.cc:492] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:17.057885 7467 ts_tablet_manager.cc:1403] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.257s user 0.012s sys 0.000s
I20260317 21:19:17.075191 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9967a6acfcf84b2c9759c612f25702e6 [id=790dbba0c4ed469a8d5e208209cb5f39]
I20260317 21:19:17.076875 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0]
I20260317 21:19:17.092780 8359 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:42830 (request call id 328) took 2653 ms. Trace:
I20260317 21:19:17.093247 8359 rpcz_store.cc:276] 0317 21:19:14.439863 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.439994 (+ 131us) service_pool.cc:225] Handling call
0317 21:19:16.535774 (+2095780us) ts_tablet_manager.cc:1151] Deleting tablet 5b066a6a5c864beb8da59bd97bacd309
0317 21:19:16.621911 (+ 86137us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:16.831931 (+210020us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:16.978313 (+146382us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:17.092699 (+114386us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2095166}
I20260317 21:19:17.095175 6659 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 5b066a6a5c864beb8da59bd97bacd309 (table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]) successfully deleted
I20260317 21:19:17.052045 7459 raft_consensus.cc:385] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.099857 7459 raft_consensus.cc:740] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.102909 7459 consensus_queue.cc:260] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.105582 7459 ts_tablet_manager.cc:1434] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.187s user 0.005s sys 0.002s
I20260317 21:19:17.105494 8454 raft_consensus.cc:493] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.105998 8454 raft_consensus.cc:515] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.108943 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3b5825c77ac4d69a39978932051db8c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:17.109985 7821 leader_election.cc:336] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779): Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.699s (SENT)
W20260317 21:19:17.110874 7261 leader_election.cc:343] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.112171 8454 leader_election.cc:290] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:17.099632 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48]
I20260317 21:19:17.124017 7462 tablet_bootstrap.cc:492] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:17.124627 7462 ts_tablet_manager.cc:1403] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.203s user 0.010s sys 0.002s
I20260317 21:19:17.127100 7462 raft_consensus.cc:359] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.127511 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]
I20260317 21:19:17.129136 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_39092d36aadb42628dabdb2bf48c1950 [id=831b6f16d49a402e8cc3d53234e050ba]
I20260317 21:19:17.130707 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994]
I20260317 21:19:17.133031 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_38fe3ad6f1bf4796af58d99b53c46560 [id=872cbb89b0d1405e9858d37dc22c05c4]
I20260317 21:19:17.133771 7462 raft_consensus.cc:385] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.134865 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d821b351490e4553b71a09c4055e26cd [id=87c45aeb0b21447bac6301a6b5845376]
I20260317 21:19:17.136343 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3]
I20260317 21:19:17.136762 7462 raft_consensus.cc:740] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.139670 7462 consensus_queue.cc:260] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.151252 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845]
I20260317 21:19:17.152774 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097]
I20260317 21:19:17.154201 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]
I20260317 21:19:17.155458 7462 ts_tablet_manager.cc:1434] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.030s user 0.005s sys 0.000s
I20260317 21:19:17.156855 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3]
I20260317 21:19:17.158444 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552]
I20260317 21:19:17.160367 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_08bd63c86aca4345bc3ca2746d45456e [id=93623d5786f645c7a1ad496e656caa24]
I20260317 21:19:17.162627 8456 raft_consensus.cc:515] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.061059 7467 raft_consensus.cc:359] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.163763 7467 raft_consensus.cc:385] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.165977 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2dccb915fad4666bd7978f8c416db44" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:17.164050 7467 raft_consensus.cc:740] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.168081 7467 consensus_queue.cc:260] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.169028 7179 raft_consensus.cc:3060] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:17.169592 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]
I20260317 21:19:17.171066 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96]
I20260317 21:19:17.172490 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0]
I20260317 21:19:17.173903 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0]
I20260317 21:19:17.183045 7179 raft_consensus.cc:2468] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:17.185786 7261 leader_election.cc:304] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:17.186627 8454 raft_consensus.cc:2804] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:17.187098 8454 raft_consensus.cc:697] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:17.188930 8454 consensus_queue.cc:237] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.192000 7462 tablet_bootstrap.cc:492] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:17.154006 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3b5825c77ac4d69a39978932051db8c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:17.203960 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2dccb915fad4666bd7978f8c416db44" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:17.163137 8383 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.204555 7037 raft_consensus.cc:3060] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:17.204713 8383 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.207782 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:17.209175 6971 leader_election.cc:343] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.210363 8383 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:17.212059 7037 raft_consensus.cc:2468] T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
W20260317 21:19:17.235534 6971 consensus_peers.cc:597] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:17.240437 7462 tablet_bootstrap.cc:654] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:17.243561 7459 tablet_bootstrap.cc:492] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:17.244213 8453 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.247521 8453 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.250059 8453 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:17.251679 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:17.253741 7261 leader_election.cc:343] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:17.260535 7264 leader_election.cc:343] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.261016 7264 leader_election.cc:304] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:17.262233 8453 raft_consensus.cc:2749] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:17.264155 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:17.264791 7328 raft_consensus.cc:2468] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:17.184206 8456 leader_election.cc:290] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:17.287122 7462 tablet_bootstrap.cc:492] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:17.287581 7462 ts_tablet_manager.cc:1403] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.096s user 0.010s sys 0.000s
I20260317 21:19:17.290839 7462 raft_consensus.cc:359] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.291476 7462 raft_consensus.cc:385] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.291785 7462 raft_consensus.cc:740] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.292465 7462 consensus_queue.cc:260] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.310149 7462 ts_tablet_manager.cc:1434] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260317 21:19:17.316274 7462 tablet_bootstrap.cc:492] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:17.363356 7462 tablet_bootstrap.cc:654] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:17.306089 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]
I20260317 21:19:17.381901 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:17.382460 7036 raft_consensus.cc:2468] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:17.383620 7462 tablet_bootstrap.cc:492] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:17.384016 7462 ts_tablet_manager.cc:1403] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.000s
I20260317 21:19:17.386250 6973 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:17.387279 8435 raft_consensus.cc:2804] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:17.387650 8435 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:17.388033 8435 raft_consensus.cc:3060] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:17.395614 8435 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.400391 7462 raft_consensus.cc:359] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.401089 7462 raft_consensus.cc:385] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.401343 7462 raft_consensus.cc:740] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.402298 7467 ts_tablet_manager.cc:1434] T b2dccb915fad4666bd7978f8c416db44 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.344s user 0.006s sys 0.000s
I20260317 21:19:17.409121 7462 consensus_queue.cc:260] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.410880 7462 ts_tablet_manager.cc:1434] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.027s user 0.007s sys 0.000s
I20260317 21:19:17.411775 7462 tablet_bootstrap.cc:492] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:17.370213 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7fee5e45b4aa4a93900162f41e3a3d60 [id=a73a2c0e2c814324ba832290ee403c4a]
I20260317 21:19:17.413404 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_6254d435474446bca35e4c78478b186f [id=a796a06b6825482b969b97c9da0ab79d]
I20260317 21:19:17.415594 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_2cd62a2d2daa4868a58114c426bca181 [id=ab3da0a6fea343d3b2d399b3437fe6f1]
I20260317 21:19:17.446029 7467 tablet_bootstrap.cc:492] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:17.446606 7462 tablet_bootstrap.cc:654] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:17.474330 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139]
I20260317 21:19:17.490660 7462 tablet_bootstrap.cc:492] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:17.491083 7462 ts_tablet_manager.cc:1403] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.080s user 0.009s sys 0.000s
I20260317 21:19:17.493373 7462 raft_consensus.cc:359] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.493995 7462 raft_consensus.cc:385] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.494299 7462 raft_consensus.cc:740] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.503062 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e]
I20260317 21:19:17.504582 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541]
I20260317 21:19:17.505937 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 [id=b1d87190d48c47288ea093aa65a528a0]
I20260317 21:19:17.507201 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343]
I20260317 21:19:17.512923 7462 consensus_queue.cc:260] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.515055 7462 ts_tablet_manager.cc:1434] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260317 21:19:17.515942 7462 tablet_bootstrap.cc:492] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:17.520467 7261 consensus_peers.cc:597] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:17.558976 7462 tablet_bootstrap.cc:654] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:17.508469 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af]
I20260317 21:19:17.569095 8435 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:17.569888 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:17.579047 7467 tablet_bootstrap.cc:654] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:15.571097 7918 rpcz_store.cc:276] 0317 21:19:14.299282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:14.299715 (+ 433us) service_pool.cc:225] Handling call
0317 21:19:14.310262 (+ 10547us) raft_consensus.cc:1471] Updating replica for 110 ops
0317 21:19:14.751145 (+440883us) raft_consensus.cc:1534] Early marking committed up to index 312
0317 21:19:14.751815 (+ 670us) raft_consensus.cc:1539] Triggering prepare for 110 ops
0317 21:19:14.752190 (+ 375us) write_op.cc:270] Start()
0317 21:19:14.908945 (+156755us) write_op.cc:276] Timestamp: P: 1773782344737401 usec, L: 0
0317 21:19:14.909633 (+ 688us) write_op.cc:270] Start()
0317 21:19:14.909723 (+ 90us) write_op.cc:276] Timestamp: P: 1773782344757184 usec, L: 0
0317 21:19:14.910193 (+ 470us) write_op.cc:270] Start()
0317 21:19:14.910265 (+ 72us) write_op.cc:276] Timestamp: P: 1773782344759169 usec, L: 0
0317 21:19:14.910743 (+ 478us) write_op.cc:270] Start()
0317 21:19:14.910808 (+ 65us) write_op.cc:276] Timestamp: P: 1773782344850340 usec, L: 0
0317 21:19:14.911220 (+ 412us) write_op.cc:270] Start()
0317 21:19:14.911289 (+ 69us) write_op.cc:276] Timestamp: P: 1773782344942909 usec, L: 0
0317 21:19:14.911753 (+ 464us) write_op.cc:270] Start()
0317 21:19:14.986787 (+ 75034us) write_op.cc:276] Timestamp: P: 1773782345025406 usec, L: 0
0317 21:19:14.987342 (+ 555us) write_op.cc:270] Start()
0317 21:19:14.987482 (+ 140us) write_op.cc:276] Timestamp: P: 1773782345162037 usec, L: 0
0317 21:19:14.987970 (+ 488us) write_op.cc:270] Start()
0317 21:19:15.064582 (+ 76612us) write_op.cc:276] Timestamp: P: 1773782345224666 usec, L: 0
0317 21:19:15.068717 (+ 4135us) write_op.cc:270] Start()
0317 21:19:15.068805 (+ 88us) write_op.cc:276] Timestamp: P: 1773782345265207 usec, L: 0
0317 21:19:15.069308 (+ 503us) write_op.cc:270] Start()
0317 21:19:15.069377 (+ 69us) write_op.cc:276] Timestamp: P: 1773782345286168 usec, L: 0
0317 21:19:15.069810 (+ 433us) write_op.cc:270] Start()
0317 21:19:15.069877 (+ 67us) write_op.cc:276] Timestamp: P: 1773782345296149 usec, L: 0
0317 21:19:15.070330 (+ 453us) write_op.cc:270] Start()
0317 21:19:15.070400 (+ 70us) write_op.cc:276] Timestamp: P: 1773782345299177 usec, L: 0
0317 21:19:15.070857 (+ 457us) write_op.cc:270] Start()
0317 21:19:15.070935 (+ 78us) write_op.cc:276] Timestamp: P: 1773782345464272 usec, L: 0
0317 21:19:15.071371 (+ 436us) write_op.cc:270] Start()
0317 21:19:15.071442 (+ 71us) write_op.cc:276] Timestamp: P: 1773782345515102 usec, L: 0
0317 21:19:15.071876 (+ 434us) write_op.cc:270] Start()
0317 21:19:15.071968 (+ 92us) write_op.cc:276] Timestamp: P: 1773782345640394 usec, L: 0
0317 21:19:15.147368 (+ 75400us) write_op.cc:270] Start()
0317 21:19:15.147483 (+ 115us) write_op.cc:276] Timestamp: P: 1773782345777507 usec, L: 0
0317 21:19:15.148054 (+ 571us) write_op.cc:270] Start()
0317 21:19:15.148126 (+ 72us) write_op.cc:276] Timestamp: P: 1773782345954685 usec, L: 0
0317 21:19:15.148568 (+ 442us) write_op.cc:270] Start()
0317 21:19:15.148635 (+ 67us) write_op.cc:276] Timestamp: P: 1773782346267423 usec, L: 0
0317 21:19:15.149102 (+ 467us) write_op.cc:270] Start()
0317 21:19:15.149169 (+ 67us) write_op.cc:276] Timestamp: P: 1773782346275401 usec, L: 0
0317 21:19:15.149570 (+ 401us) write_op.cc:270] Start()
0317 21:19:15.149647 (+ 77us) write_op.cc:276] Timestamp: P: 1773782346341971 usec, L: 0
0317 21:19:15.150018 (+ 371us) write_op.cc:270] Start()
0317 21:19:15.150097 (+ 79us) write_op.cc:276] Timestamp: P: 1773782346344033 usec, L: 0
0317 21:19:15.150527 (+ 430us) write_op.cc:270] Start()
0317 21:19:15.150602 (+ 75us) write_op.cc:276] Timestamp: P: 1773782346348220 usec, L: 0
0317 21:19:15.151009 (+ 407us) write_op.cc:270] Start()
0317 21:19:15.151077 (+ 68us) write_op.cc:276] Timestamp: P: 1773782346351192 usec, L: 0
0317 21:19:15.151485 (+ 408us) write_op.cc:270] Start()
0317 21:19:15.151570 (+ 85us) write_op.cc:276] Timestamp: P: 1773782346354274 usec, L: 0
0317 21:19:15.152027 (+ 457us) write_op.cc:270] Start()
0317 21:19:15.152096 (+ 69us) write_op.cc:276] Timestamp: P: 1773782346357162 usec, L: 0
0317 21:19:15.152480 (+ 384us) write_op.cc:270] Start()
0317 21:19:15.152555 (+ 75us) write_op.cc:276] Timestamp: P: 1773782346434623 usec, L: 0
0317 21:19:15.210691 (+ 58136us) write_op.cc:270] Start()
0317 21:19:15.210783 (+ 92us) write_op.cc:276] Timestamp: P: 1773782346653787 usec, L: 0
0317 21:19:15.211264 (+ 481us) write_op.cc:270] Start()
0317 21:19:15.211354 (+ 90us) write_op.cc:276] Timestamp: P: 1773782346657541 usec, L: 0
0317 21:19:15.211762 (+ 408us) write_op.cc:270] Start()
0317 21:19:15.211839 (+ 77us) write_op.cc:276] Timestamp: P: 1773782346779934 usec, L: 0
0317 21:19:15.212265 (+ 426us) write_op.cc:270] Start()
0317 21:19:15.212339 (+ 74us) write_op.cc:276] Timestamp: P: 1773782346788455 usec, L: 0
0317 21:19:15.212813 (+ 474us) write_op.cc:270] Start()
0317 21:19:15.212887 (+ 74us) write_op.cc:276] Timestamp: P: 1773782346791641 usec, L: 0
0317 21:19:15.213309 (+ 422us) write_op.cc:270] Start()
0317 21:19:15.215256 (+ 1947us) write_op.cc:276] Timestamp: P: 1773782346795610 usec, L: 0
0317 21:19:15.217380 (+ 2124us) write_op.cc:270] Start()
0317 21:19:15.217465 (+ 85us) write_op.cc:276] Timestamp: P: 1773782346912352 usec, L: 0
0317 21:19:15.218034 (+ 569us) write_op.cc:270] Start()
0317 21:19:15.218185 (+ 151us) write_op.cc:276] Timestamp: P: 1773782346917123 usec, L: 0
0317 21:19:15.218690 (+ 505us) write_op.cc:270] Start()
0317 21:19:15.218771 (+ 81us) write_op.cc:276] Timestamp: P: 1773782347056031 usec, L: 0
0317 21:19:15.219177 (+ 406us) write_op.cc:270] Start()
0317 21:19:15.219245 (+ 68us) write_op.cc:276] Timestamp: P: 1773782347059367 usec, L: 0
0317 21:19:15.219631 (+ 386us) write_op.cc:270] Start()
0317 21:19:15.219697 (+ 66us) write_op.cc:276] Timestamp: P: 1773782347199549 usec, L: 0
0317 21:19:15.220068 (+ 371us) write_op.cc:270] Start()
0317 21:19:15.220134 (+ 66us) write_op.cc:276] Timestamp: P: 1773782347226492 usec, L: 0
0317 21:19:15.220549 (+ 415us) write_op.cc:270] Start()
0317 21:19:15.220631 (+ 82us) write_op.cc:276] Timestamp: P: 1773782347309388 usec, L: 0
0317 21:19:15.221070 (+ 439us) write_op.cc:270] Start()
0317 21:19:15.221151 (+ 81us) write_op.cc:276] Timestamp: P: 1773782347350148 usec, L: 0
0317 21:19:15.221573 (+ 422us) write_op.cc:270] Start()
0317 21:19:15.221642 (+ 69us) write_op.cc:276] Timestamp: P: 1773782347546845 usec, L: 0
0317 21:19:15.222050 (+ 408us) write_op.cc:270] Start()
0317 21:19:15.222119 (+ 69us) write_op.cc:276] Timestamp: P: 1773782347622563 usec, L: 0
0317 21:19:15.222557 (+ 438us) write_op.cc:270] Start()
0317 21:19:15.222630 (+ 73us) write_op.cc:276] Timestamp: P: 1773782347643679 usec, L: 0
0317 21:19:15.223013 (+ 383us) write_op.cc:270] Start()
0317 21:19:15.223081 (+ 68us) write_op.cc:276] Timestamp: P: 1773782347680032 usec, L: 0
0317 21:19:15.230855 (+ 7774us) write_op.cc:270] Start()
0317 21:19:15.230942 (+ 87us) write_op.cc:276] Timestamp: P: 1773782347686074 usec, L: 0
0317 21:19:15.231404 (+ 462us) write_op.cc:270] Start()
0317 21:19:15.231472 (+ 68us) write_op.cc:276] Timestamp: P: 1773782347700922 usec, L: 0
0317 21:19:15.231885 (+ 413us) write_op.cc:270] Start()
0317 21:19:15.231958 (+ 73us) write_op.cc:276] Timestamp: P: 1773782347726965 usec, L: 0
0317 21:19:15.232353 (+ 395us) write_op.cc:270] Start()
0317 21:19:15.232419 (+ 66us) write_op.cc:276] Timestamp: P: 1773782347776727 usec, L: 0
0317 21:19:15.248458 (+ 16039us) write_op.cc:270] Start()
0317 21:19:15.248568 (+ 110us) write_op.cc:276] Timestamp: P: 1773782347845735 usec, L: 0
0317 21:19:15.249109 (+ 541us) write_op.cc:270] Start()
0317 21:19:15.249189 (+ 80us) write_op.cc:276] Timestamp: P: 1773782347848077 usec, L: 0
0317 21:19:15.249605 (+ 416us) write_op.cc:270] Start()
0317 21:19:15.249681 (+ 76us) write_op.cc:276] Timestamp: P: 1773782347854370 usec, L: 0
0317 21:19:15.250067 (+ 386us) write_op.cc:270] Start()
0317 21:19:15.250134 (+ 67us) write_op.cc:276] Timestamp: P: 1773782347856492 usec, L: 0
0317 21:19:15.250520 (+ 386us) write_op.cc:270] Start()
0317 21:19:15.250611 (+ 91us) write_op.cc:276] Timestamp: P: 1773782347966449 usec, L: 0
0317 21:19:15.251033 (+ 422us) write_op.cc:270] Start()
0317 21:19:15.251106 (+ 73us) write_op.cc:276] Timestamp: P: 1773782347968295 usec, L: 0
0317 21:19:15.251532 (+ 426us) write_op.cc:270] Start()
0317 21:19:15.251621 (+ 89us) write_op.cc:276] Timestamp: P: 1773782348042132 usec, L: 0
0317 21:19:15.252071 (+ 450us) write_op.cc:270] Start()
0317 21:19:15.252148 (+ 77us) write_op.cc:276] Timestamp: P: 1773782348043996 usec, L: 0
0317 21:19:15.252621 (+ 473us) write_op.cc:270] Start()
0317 21:19:15.280211 (+ 27590us) write_op.cc:276] Timestamp: P: 1773782348156603 usec, L: 0
0317 21:19:15.280823 (+ 612us) write_op.cc:270] Start()
0317 21:19:15.280910 (+ 87us) write_op.cc:276] Timestamp: P: 1773782348206792 usec, L: 0
0317 21:19:15.281398 (+ 488us) write_op.cc:270] Start()
0317 21:19:15.281480 (+ 82us) write_op.cc:276] Timestamp: P: 1773782348209121 usec, L: 0
0317 21:19:15.281933 (+ 453us) write_op.cc:270] Start()
0317 21:19:15.282079 (+ 146us) write_op.cc:276] Timestamp: P: 1773782348426150 usec, L: 0
0317 21:19:15.282510 (+ 431us) write_op.cc:270] Start()
0317 21:19:15.282581 (+ 71us) write_op.cc:276] Timestamp: P: 1773782348428769 usec, L: 0
0317 21:19:15.282978 (+ 397us) write_op.cc:270] Start()
0317 21:19:15.283048 (+ 70us) write_op.cc:276] Timestamp: P: 1773782348565096 usec, L: 0
0317 21:19:15.283450 (+ 402us) write_op.cc:270] Start()
0317 21:19:15.283519 (+ 69us) write_op.cc:276] Timestamp: P: 1773782348589399 usec, L: 0
0317 21:19:15.283944 (+ 425us) write_op.cc:270] Start()
0317 21:19:15.284037 (+ 93us) write_op.cc:276] Timestamp: P: 1773782348591749 usec, L: 0
0317 21:19:15.284550 (+ 513us) write_op.cc:270] Start()
0317 21:19:15.284631 (+ 81us) write_op.cc:276] Timestamp: P: 1773782348598294 usec, L: 0
0317 21:19:15.301391 (+ 16760us) write_op.cc:270] Start()
0317 21:19:15.301488 (+ 97us) write_op.cc:276] Timestamp: P: 1773782348602589 usec, L: 0
0317 21:19:15.301995 (+ 507us) write_op.cc:270] Start()
0317 21:19:15.302075 (+ 80us) write_op.cc:276] Timestamp: P: 1773782348622068 usec, L: 0
0317 21:19:15.302479 (+ 404us) write_op.cc:270] Start()
0317 21:19:15.302565 (+ 86us) write_op.cc:276] Timestamp: P: 1773782348832958 usec, L: 0
0317 21:19:15.302996 (+ 431us) write_op.cc:270] Start()
0317 21:19:15.303070 (+ 74us) write_op.cc:276] Timestamp: P: 1773782348867927 usec, L: 0
0317 21:19:15.303475 (+ 405us) write_op.cc:270] Start()
0317 21:19:15.303548 (+ 73us) write_op.cc:276] Timestamp: P: 1773782348942395 usec, L: 0
0317 21:19:15.303997 (+ 449us) write_op.cc:270] Start()
0317 21:19:15.304063 (+ 66us) write_op.cc:276] Timestamp: P: 1773782348976781 usec, L: 0
0317 21:19:15.304489 (+ 426us) write_op.cc:270] Start()
0317 21:19:15.304575 (+ 86us) write_op.cc:276] Timestamp: P: 1773782349188367 usec, L: 0
0317 21:19:15.331347 (+ 26772us) write_op.cc:270] Start()
0317 21:19:15.331433 (+ 86us) write_op.cc:276] Timestamp: P: 1773782349191642 usec, L: 0
0317 21:19:15.331918 (+ 485us) write_op.cc:270] Start()
0317 21:19:15.332002 (+ 84us) write_op.cc:276] Timestamp: P: 1773782349305344 usec, L: 0
0317 21:19:15.332381 (+ 379us) write_op.cc:270] Start()
0317 21:19:15.338756 (+ 6375us) write_op.cc:276] Timestamp: P: 1773782349313090 usec, L: 0
0317 21:19:15.339386 (+ 630us) write_op.cc:270] Start()
0317 21:19:15.339476 (+ 90us) write_op.cc:276] Timestamp: P: 1773782349471848 usec, L: 0
0317 21:19:15.339907 (+ 431us) write_op.cc:270] Start()
0317 21:19:15.339973 (+ 66us) write_op.cc:276] Timestamp: P: 1773782349534682 usec, L: 0
0317 21:19:15.340386 (+ 413us) write_op.cc:270] Start()
0317 21:19:15.340457 (+ 71us) write_op.cc:276] Timestamp: P: 1773782349576040 usec, L: 0
0317 21:19:15.340944 (+ 487us) write_op.cc:270] Start()
0317 21:19:15.341014 (+ 70us) write_op.cc:276] Timestamp: P: 1773782349578847 usec, L: 0
0317 21:19:15.341410 (+ 396us) write_op.cc:270] Start()
0317 21:19:15.341499 (+ 89us) write_op.cc:276] Timestamp: P: 1773782349606126 usec, L: 0
0317 21:19:15.343999 (+ 2500us) write_op.cc:270] Start()
0317 21:19:15.344083 (+ 84us) write_op.cc:276] Timestamp: P: 1773782349621702 usec, L: 0
0317 21:19:15.344596 (+ 513us) write_op.cc:270] Start()
0317 21:19:15.344700 (+ 104us) write_op.cc:276] Timestamp: P: 1773782349704416 usec, L: 0
0317 21:19:15.345111 (+ 411us) write_op.cc:270] Start()
0317 21:19:15.345191 (+ 80us) write_op.cc:276] Timestamp: P: 1773782349706586 usec, L: 0
0317 21:19:15.345567 (+ 376us) write_op.cc:270] Start()
0317 21:19:15.345638 (+ 71us) write_op.cc:276] Timestamp: P: 1773782350087182 usec, L: 0
0317 21:19:15.346017 (+ 379us) write_op.cc:270] Start()
0317 21:19:15.346079 (+ 62us) write_op.cc:276] Timestamp: P: 1773782350217188 usec, L: 0
0317 21:19:15.346443 (+ 364us) write_op.cc:270] Start()
0317 21:19:15.346517 (+ 74us) write_op.cc:276] Timestamp: P: 1773782350258524 usec, L: 0
0317 21:19:15.346884 (+ 367us) write_op.cc:270] Start()
0317 21:19:15.346948 (+ 64us) write_op.cc:276] Timestamp: P: 1773782350287349 usec, L: 0
0317 21:19:15.347312 (+ 364us) write_op.cc:270] Start()
0317 21:19:15.347384 (+ 72us) write_op.cc:276] Timestamp: P: 1773782350290470 usec, L: 0
0317 21:19:15.347810 (+ 426us) write_op.cc:270] Start()
0317 21:19:15.347880 (+ 70us) write_op.cc:276] Timestamp: P: 1773782350392382 usec, L: 0
0317 21:19:15.348273 (+ 393us) write_op.cc:270] Start()
0317 21:19:15.348355 (+ 82us) write_op.cc:276] Timestamp: P: 1773782350633661 usec, L: 0
0317 21:19:15.348813 (+ 458us) write_op.cc:270] Start()
0317 21:19:15.348885 (+ 72us) write_op.cc:276] Timestamp: P: 1773782350637232 usec, L: 0
0317 21:19:15.349277 (+ 392us) write_op.cc:270] Start()
0317 21:19:15.349348 (+ 71us) write_op.cc:276] Timestamp: P: 1773782350671967 usec, L: 0
0317 21:19:15.349745 (+ 397us) write_op.cc:270] Start()
0317 21:19:15.349818 (+ 73us) write_op.cc:276] Timestamp: P: 1773782350687439 usec, L: 0
0317 21:19:15.371491 (+ 21673us) write_op.cc:270] Start()
0317 21:19:15.371627 (+ 136us) write_op.cc:276] Timestamp: P: 1773782350705811 usec, L: 0
0317 21:19:15.372153 (+ 526us) write_op.cc:270] Start()
0317 21:19:15.372245 (+ 92us) write_op.cc:276] Timestamp: P: 1773782350878201 usec, L: 0
0317 21:19:15.372877 (+ 632us) write_op.cc:270] Start()
0317 21:19:15.372978 (+ 101us) write_op.cc:276] Timestamp: P: 1773782350894943 usec, L: 0
0317 21:19:15.373478 (+ 500us) write_op.cc:270] Start()
0317 21:19:15.373656 (+ 178us) write_op.cc:276] Timestamp: P: 1773782350920388 usec, L: 0
0317 21:19:15.380835 (+ 7179us) write_op.cc:270] Start()
0317 21:19:15.380939 (+ 104us) write_op.cc:276] Timestamp: P: 1773782350965161 usec, L: 0
0317 21:19:15.381391 (+ 452us) write_op.cc:270] Start()
0317 21:19:15.381459 (+ 68us) write_op.cc:276] Timestamp: P: 1773782351124733 usec, L: 0
0317 21:19:15.381893 (+ 434us) write_op.cc:270] Start()
0317 21:19:15.381964 (+ 71us) write_op.cc:276] Timestamp: P: 1773782351140695 usec, L: 0
0317 21:19:15.382343 (+ 379us) write_op.cc:270] Start()
0317 21:19:15.382407 (+ 64us) write_op.cc:276] Timestamp: P: 1773782351203956 usec, L: 0
0317 21:19:15.382788 (+ 381us) write_op.cc:270] Start()
0317 21:19:15.382854 (+ 66us) write_op.cc:276] Timestamp: P: 1773782351218895 usec, L: 0
0317 21:19:15.383238 (+ 384us) write_op.cc:270] Start()
0317 21:19:15.383303 (+ 65us) write_op.cc:276] Timestamp: P: 1773782351328915 usec, L: 0
0317 21:19:15.383702 (+ 399us) write_op.cc:270] Start()
0317 21:19:15.383772 (+ 70us) write_op.cc:276] Timestamp: P: 1773782351371225 usec, L: 0
0317 21:19:15.384141 (+ 369us) write_op.cc:270] Start()
0317 21:19:15.384202 (+ 61us) write_op.cc:276] Timestamp: P: 1773782351375602 usec, L: 0
0317 21:19:15.384615 (+ 413us) write_op.cc:270] Start()
0317 21:19:15.394043 (+ 9428us) write_op.cc:276] Timestamp: P: 1773782351378605 usec, L: 0
0317 21:19:15.394625 (+ 582us) write_op.cc:270] Start()
0317 21:19:15.394704 (+ 79us) write_op.cc:276] Timestamp: P: 1773782351463447 usec, L: 0
0317 21:19:15.395152 (+ 448us) write_op.cc:270] Start()
0317 21:19:15.395217 (+ 65us) write_op.cc:276] Timestamp: P: 1773782351760285 usec, L: 0
0317 21:19:15.513419 (+118202us) log.cc:844] Serialized 69846 byte log entry
0317 21:19:15.513971 (+ 552us) raft_consensus.cc:1645] Marking committed up to 419
0317 21:19:15.525658 (+ 11687us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.525685 (+ 27us) write_op.cc:454] Released schema lock
0317 21:19:15.527735 (+ 2050us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.527756 (+ 21us) write_op.cc:454] Released schema lock
0317 21:19:15.531169 (+ 3413us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.531191 (+ 22us) write_op.cc:454] Released schema lock
0317 21:19:15.533139 (+ 1948us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.533163 (+ 24us) write_op.cc:454] Released schema lock
0317 21:19:15.570139 (+ 36976us) raft_consensus.cc:1697] Filling consensus response to leader.
0317 21:19:15.570226 (+ 87us) raft_consensus.cc:1668] Waiting on the replicates to finish logging
0317 21:19:15.570268 (+ 42us) raft_consensus.cc:1685] finished
0317 21:19:15.570281 (+ 13us) raft_consensus.cc:1691] UpdateReplicas() finished
0317 21:19:15.570679 (+ 398us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:15.468287 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.468688 (+ 401us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.468711 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:19:15.468724 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:15.468988 (+ 264us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.469038 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.469058 (+ 20us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:15.469234 (+ 176us) tablet.cc:700] Row locks acquired
0317 21:19:15.469246 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:15.514191 (+ 44945us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.514791 (+ 600us) write_op.cc:301] APPLY: starting
0317 21:19:15.514900 (+ 109us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.515121 (+ 221us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.515181 (+ 60us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.516348 (+ 1167us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.516614 (+ 266us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:15.516909 (+ 295us) write_op.cc:312] APPLY: finished
0317 21:19:15.517221 (+ 312us) log.cc:844] Serialized 29 byte log entry
0317 21:19:15.519801 (+ 2580us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.520062 (+ 261us) write_op.cc:454] Released schema lock
0317 21:19:15.520098 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:15.520492 (+ 394us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.520545 (+ 53us) write_op.cc:454] Released schema lock
Related trace 'op':
0317 21:19:15.469448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.469947 (+ 499us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.469970 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:19:15.469982 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:15.470219 (+ 237us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.470266 (+ 47us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.470288 (+ 22us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:15.470446 (+ 158us) tablet.cc:700] Row locks acquired
0317 21:19:15.470459 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:15.521711 (+ 51252us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.522157 (+ 446us) write_op.cc:301] APPLY: starting
0317 21:19:15.522218 (+ 61us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.522318 (+ 100us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.522334 (+ 16us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.522775 (+ 441us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.522930 (+ 155us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:15.523053 (+ 123us) write_op.cc:312] APPLY: finished
0317 21:19:15.523245 (+ 192us) log.cc:844] Serialized 29 byte log entry
0317 21:19:15.523546 (+ 301us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.523605 (+ 59us) write_op.cc:454] Released schema lock
0317 21:19:15.523641 (+ 36us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:15.523839 (+ 198us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.523862 (+ 23us) write_op.cc:454] Released schema lock
Related trace 'op':
0317 21:19:15.504908 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.505352 (+ 444us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.505382 (+ 30us) write_op.cc:435] Acquired schema lock
0317 21:19:15.505397 (+ 15us) tablet.cc:661] Decoding operations
0317 21:19:15.505717 (+ 320us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.505772 (+ 55us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.506034 (+ 262us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:15.506262 (+ 228us) tablet.cc:700] Row locks acquired
0317 21:19:15.506277 (+ 15us) write_op.cc:260] PREPARE: finished
0317 21:19:15.521961 (+ 15684us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.523864 (+ 1903us) write_op.cc:301] APPLY: starting
0317 21:19:15.523944 (+ 80us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.524110 (+ 166us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.524159 (+ 49us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.525211 (+ 1052us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.525475 (+ 264us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:15.525703 (+ 228us) write_op.cc:312] APPLY: finished
0317 21:19:15.526023 (+ 320us) log.cc:844] Serialized 29 byte log entry
0317 21:19:15.526576 (+ 553us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.526709 (+ 133us) write_op.cc:454] Released schema lock
0317 21:19:15.526778 (+ 69us) write_op.cc:341] FINISH: Updating metrics
0317 21:19:15.527083 (+ 305us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.527148 (+ 65us) write_op.cc:454] Released schema lock
Related trace 'op':
0317 21:19:15.506623 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.516416 (+ 9793us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.516440 (+ 24us) write_op.cc:435] Acquired schema lock
0317 21:19:15.516453 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:15.516746 (+ 293us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.516799 (+ 53us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.516820 (+ 21us) tablet.cc:684] Acquiring locks for 1 operations
0317 21:19:15.516959 (+ 139us) tablet.cc:700] Row locks acquired
0317 21:19:15.516973 (+ 14us) write_op.cc:260] PREPARE: finished
0317 21:19:15.524159 (+ 7186us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.524357 (+ 198us) write_op.cc:301] APPLY: starting
0317 21:19:15.524401 (+ 44us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.524519 (+ 118us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.524545 (+ 26us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.524691 (+ 146us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.524830 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:15.524961 (+ 131us) write_op.cc:312] APPLY: finished
0317 21:19:15.525147 (+ 186us) log.cc:844] Serialized 23 byte log entry
0317 21:19:15.525267 (+ 120us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.525323 (+ 56us) write_op.cc:454] Released schema lock
0317 21:19:15.525353 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0317 21:19:15.517197 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.517552 (+ 355us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.517573 (+ 21us) write_op.cc:435] Acquired schema lock
0317 21:19:15.517587 (+ 14us) tablet.cc:661] Decoding operations
0317 21:19:15.517853 (+ 266us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.517899 (+ 46us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.517923 (+ 24us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:15.518101 (+ 178us) tablet.cc:700] Row locks acquired
0317 21:19:15.518113 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:15.525923 (+ 7810us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.526134 (+ 211us) write_op.cc:301] APPLY: starting
0317 21:19:15.526186 (+ 52us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.526284 (+ 98us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.526306 (+ 22us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.526542 (+ 236us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.526681 (+ 139us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:15.526801 (+ 120us) write_op.cc:312] APPLY: finished
0317 21:19:15.526964 (+ 163us) log.cc:844] Serialized 29 byte log entry
0317 21:19:15.527393 (+ 429us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.527456 (+ 63us) write_op.cc:454] Released schema lock
0317 21:19:15.527487 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0317 21:19:15.518309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.518649 (+ 340us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.518668 (+ 19us) write_op.cc:435] Acquired schema lock
0317 21:19:15.518680 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:15.518940 (+ 260us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.518990 (+ 50us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.519017 (+ 27us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:15.519185 (+ 168us) tablet.cc:700] Row locks acquired
0317 21:19:15.519200 (+ 15us) write_op.cc:260] PREPARE: finished
0317 21:19:15.527956 (+ 8756us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.529548 (+ 1592us) write_op.cc:301] APPLY: starting
0317 21:19:15.529596 (+ 48us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.529699 (+ 103us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.529713 (+ 14us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.530023 (+ 310us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.530238 (+ 215us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0317 21:19:15.530347 (+ 109us) write_op.cc:312] APPLY: finished
0317 21:19:15.530538 (+ 191us) log.cc:844] Serialized 29 byte log entry
0317 21:19:15.530812 (+ 274us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:15.530874 (+ 62us) write_op.cc:454] Released schema lock
0317 21:19:15.530904 (+ 30us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0317 21:19:15.519393 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:15.519734 (+ 341us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:15.519757 (+ 23us) write_op.cc:435] Acquired schema lock
0317 21:19:15.519773 (+ 16us) tablet.cc:661] Decoding operations
0317 21:19:15.520034 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:15.520091 (+ 57us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:15.520114 (+ 23us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:15.520339 (+ 225us) tablet.cc:700] Row locks acquired
0317 21:19:15.520352 (+ 13us) write_op.cc:260] PREPARE: finished
0317 21:19:15.531437 (+ 11085us) op_driver.cc:464] REPLICATION: finished
0317 21:19:15.531656 (+ 219us) write_op.cc:301] APPLY: starting
0317 21:19:15.531699 (+ 43us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:15.531798 (+ 99us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:15.531816 (+ 18us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:15.532035 (+ 219us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:15.532166 (+ 131us) tablet_metrics.cc:563]
I20260317 21:19:17.540849 7459 tablet_bootstrap.cc:654] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:17.506453 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 120269us
W20260317 21:19:17.571286 6971 leader_election.cc:343] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.605702 7264 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:17.607700 8454 raft_consensus.cc:2804] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:17.608076 8454 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:17.608392 8454 raft_consensus.cc:3060] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:17.627595 7462 tablet_bootstrap.cc:492] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:17.628029 7462 ts_tablet_manager.cc:1403] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.112s user 0.009s sys 0.000s
I20260317 21:19:17.628340 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:17.630877 8383 consensus_queue.cc:1048] T e586f11c37184506a15d726b4eb333be P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:17.632577 7462 raft_consensus.cc:359] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.633281 7462 raft_consensus.cc:385] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.633543 7462 raft_consensus.cc:740] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.634152 7462 consensus_queue.cc:260] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.636265 7462 ts_tablet_manager.cc:1434] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260317 21:19:17.615828 8454 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.683552 7328 raft_consensus.cc:2393] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:17.645490 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436]
I20260317 21:19:17.687639 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b]
I20260317 21:19:17.689983 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d4a5487a2b7e4218b41223f155418e27 [id=b816766f31714663b685cfcc9a9f537a]
I20260317 21:19:17.691437 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7]
I20260317 21:19:17.693591 8383 raft_consensus.cc:493] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.694119 8383 raft_consensus.cc:515] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.696920 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:17.698787 6971 leader_election.cc:343] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.701751 8456 raft_consensus.cc:493] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.702472 8456 raft_consensus.cc:515] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.704454 8456 leader_election.cc:290] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:17.710686 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:17.712275 7261 leader_election.cc:343] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.714740 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:17.716181 7261 leader_election.cc:343] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:17.722556 7459 tablet_bootstrap.cc:492] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:17.723111 7459 ts_tablet_manager.cc:1403] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.480s user 0.006s sys 0.005s
I20260317 21:19:17.726092 7459 raft_consensus.cc:359] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.727857 8456 consensus_queue.cc:1048] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:17.730933 8487 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:17.731561 8487 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
W20260317 21:19:17.745339 7261 consensus_peers.cc:597] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:17.752107 8492 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.752754 8492 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:17.755657 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 7 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:17.756286 6683 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 47e1c983d0224428a9a7b147a0e5e4ef in term 5.
I20260317 21:19:17.757716 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 7 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:19:17.760036 8454 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:17.771474 8492 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 7 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:17.774461 8492 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Leader pre-election won for term 7
I20260317 21:19:17.775779 8492 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:17.776912 8492 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 6 FOLLOWER]: Advancing to term 7
I20260317 21:19:17.782831 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]
I20260317 21:19:17.798817 7467 tablet_bootstrap.cc:492] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:17.799369 7467 ts_tablet_manager.cc:1403] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.354s user 0.012s sys 0.000s
I20260317 21:19:17.801787 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f]
I20260317 21:19:17.802474 7459 raft_consensus.cc:385] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.802886 7459 raft_consensus.cc:740] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.793933 8383 leader_election.cc:290] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:17.817883 8494 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 5 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:17.818614 8494 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
W20260317 21:19:17.820830 7821 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.126:40107, user_credentials={real_user=slave}} blocked reactor thread for 63210.5us
I20260317 21:19:17.820348 7459 consensus_queue.cc:260] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.836889 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:17.838234 7459 ts_tablet_manager.cc:1434] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.115s user 0.006s sys 0.000s
I20260317 21:19:17.829823 7467 raft_consensus.cc:359] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.839097 8487 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:17.846628 7467 raft_consensus.cc:385] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.844666 6973 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:17.848726 8384 raft_consensus.cc:2749] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:17.849095 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_cafc44dd71714a528bda6622dac7997e [id=c11557d0407d4bf08dac964b24545e44]
I20260317 21:19:17.848598 7467 raft_consensus.cc:740] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.851342 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690]
I20260317 21:19:17.853116 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07]
I20260317 21:19:17.847604 8492 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 7 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:17.858027 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2]
I20260317 21:19:17.858352 7918 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 7 FOLLOWER]: Rejecting Update request from peer 21186091f2c9410ea6e585c5ef2c6291 for earlier term 5. Current term is 7. Ops: []
I20260317 21:19:17.860431 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32]
I20260317 21:19:17.862846 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696]
I20260317 21:19:17.874639 7462 tablet_bootstrap.cc:492] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:17.882989 7462 tablet_bootstrap.cc:654] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:17.856441 7459 tablet_bootstrap.cc:492] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:17.891013 8439 consensus_queue.cc:1048] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:17.864465 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857]
I20260317 21:19:17.856604 7467 consensus_queue.cc:260] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.895694 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]
I20260317 21:19:17.896147 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 7 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:19:17.897843 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82]
I20260317 21:19:17.887844 8488 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 }, Status: INVALID_TERM, Last received: 5.422, Next index: 423, Last known committed idx: 419, Time since last communication: 0.013s
I20260317 21:19:17.908743 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa]
I20260317 21:19:17.910434 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b]
I20260317 21:19:17.920822 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065]
I20260317 21:19:17.921172 8492 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 7 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:17.935662 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]
I20260317 21:19:17.943186 8384 consensus_queue.cc:1048] T 203f33157a0d4b82aab44da452aa46dd P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260317 21:19:17.949801 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 97588us
I20260317 21:19:17.949473 8495 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 LEADER]: Stepping down as leader of term 5
I20260317 21:19:17.943635 7467 ts_tablet_manager.cc:1434] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.144s user 0.009s sys 0.002s
I20260317 21:19:17.950608 8495 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 LEADER]: Becoming Follower/Learner. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:17.954612 7462 tablet_bootstrap.cc:492] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:17.955071 7462 ts_tablet_manager.cc:1403] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.081s user 0.015s sys 0.003s
I20260317 21:19:17.956241 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569]
I20260317 21:19:17.959379 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9c887813658144a5ab397c605d8954c0 [id=def26fdbf280448380ad71d8c52b9713]
I20260317 21:19:17.962513 7462 raft_consensus.cc:359] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.962841 7467 tablet_bootstrap.cc:492] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:17.960449 8495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 422, Committed index: 422, Last appended: 5.422, Last appended by leader: 422, Current term: 5, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:17.974251 7462 raft_consensus.cc:385] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:17.975045 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3]
I20260317 21:19:17.963155 8454 raft_consensus.cc:493] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.975419 7462 raft_consensus.cc:740] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:17.975900 8454 raft_consensus.cc:515] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.976166 7462 consensus_queue.cc:260] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.978804 7462 ts_tablet_manager.cc:1434] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.019s user 0.005s sys 0.000s
I20260317 21:19:17.978973 8454 leader_election.cc:290] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:17.979876 7462 tablet_bootstrap.cc:492] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:17.986413 8435 raft_consensus.cc:493] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:17.987005 8435 raft_consensus.cc:515] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:17.988246 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a33d113c213c48728df41846ad097ead [id=df5c9c1007aa4e84aa9363779d23864b]
I20260317 21:19:17.991075 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049]
I20260317 21:19:17.993130 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "740ae685629041f39bbbc8ce715d662e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:17.994861 7179 raft_consensus.cc:2393] T 740ae685629041f39bbbc8ce715d662e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:17.988487 8495 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 5 FOLLOWER]: Advancing to term 7
I20260317 21:19:17.996739 8435 leader_election.cc:290] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:18.011191 7467 tablet_bootstrap.cc:654] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:18.011554 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d6e9cf5141a346f7af807b9f5105eb52 [id=e8f700fb229d465ca87459fd70770227]
I20260317 21:19:18.014091 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee]
I20260317 21:19:18.015569 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf]
I20260317 21:19:18.016920 7459 tablet_bootstrap.cc:654] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:18.017258 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97]
I20260317 21:19:18.018877 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee]
I20260317 21:19:18.020347 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346]
I20260317 21:19:18.022455 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a4ca7b6127064c19b73aee408bc49282 [id=f1ff0d130d854a349d415b1d7d91a836]
I20260317 21:19:18.023917 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7864164e6f474a35a89094d36d4254ec [id=f2daa1d797d547ff9c87f31e4560581d]
I20260317 21:19:18.025492 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417]
I20260317 21:19:18.028443 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58]
I20260317 21:19:18.041826 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c]
I20260317 21:19:18.046996 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_aacec8c3d7ba49f9b6742baf37b423a8 [id=f966bc12b8824b708f7ff3d9f5789eb8]
I20260317 21:19:18.061645 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_24ec4c242298468dba3b539b6f272362 [id=fb617cbcbd634696a101417011a3a53a]
I20260317 21:19:18.063359 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_147e7f28d4344faf9b6289f6c999b6a9 [id=fd0410563367415d8e73a23c52e8e839]
I20260317 21:19:18.065136 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]
I20260317 21:19:18.067425 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71]
I20260317 21:19:18.070691 7462 tablet_bootstrap.cc:654] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:18.087579 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 564d41fc13b04ec688fc38a78f55e83d consensus state has no leader
I20260317 21:19:18.101374 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73f093d0e8434ca4c41cef6dfe310c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:18.126714 8496 raft_consensus.cc:493] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.127743 8496 raft_consensus.cc:515] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.130009 8496 leader_election.cc:290] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:18.132984 7467 tablet_bootstrap.cc:492] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:18.159663 7467 ts_tablet_manager.cc:1403] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.199s user 0.018s sys 0.005s
I20260317 21:19:18.181047 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:18.196833 8454 raft_consensus.cc:493] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.197479 8454 raft_consensus.cc:515] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.202883 7462 tablet_bootstrap.cc:492] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:18.203313 7462 ts_tablet_manager.cc:1403] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.224s user 0.013s sys 0.011s
I20260317 21:19:18.205428 8496 raft_consensus.cc:493] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.206055 8496 raft_consensus.cc:515] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.206962 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:18.208194 7037 raft_consensus.cc:2393] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f539d1a4a584b8ca83928be49369bf4 in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:18.210522 8496 leader_election.cc:290] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:18.211602 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:18.212301 7329 raft_consensus.cc:2468] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:18.214363 8292 tablet_loader.cc:96] loaded metadata for tablet 039d8bed256a43a6b98075b8430cbf3b (table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611])
I20260317 21:19:18.215682 8292 tablet_loader.cc:96] loaded metadata for tablet 047dd2548dbb4c659694b93633b39bf4 (table default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343])
I20260317 21:19:18.217263 8292 tablet_loader.cc:96] loaded metadata for tablet 054535edcd5041e9b0429c3e7248b1b7 (table default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346])
I20260317 21:19:18.219331 8292 tablet_loader.cc:96] loaded metadata for tablet 06f6a1c205a74045a9f79daf98809b45 (table default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097])
I20260317 21:19:18.238250 8292 tablet_loader.cc:96] loaded metadata for tablet 0dd4c1efc0624957832c17dc7a4db3f3 (table default.Table_d6e9cf5141a346f7af807b9f5105eb52 [id=e8f700fb229d465ca87459fd70770227])
I20260317 21:19:18.240413 8292 tablet_loader.cc:96] loaded metadata for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce])
I20260317 21:19:18.229010 7462 raft_consensus.cc:359] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.241744 8292 tablet_loader.cc:96] loaded metadata for tablet 10942327694142fa82ada834b10befab (table default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9])
I20260317 21:19:18.242877 8292 tablet_loader.cc:96] loaded metadata for tablet 1179e8f731604aabacaab3090a5f574c (table default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994])
I20260317 21:19:18.242955 7462 raft_consensus.cc:385] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:18.245311 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73f093d0e8434ca4c41cef6dfe310c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:18.246858 7462 raft_consensus.cc:740] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:18.255533 8454 leader_election.cc:290] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:18.256625 7038 raft_consensus.cc:3060] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:18.262872 8292 tablet_loader.cc:96] loaded metadata for tablet 17610bdee98f49c4b1d8b5481867c6e8 (table default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7])
I20260317 21:19:18.264078 8292 tablet_loader.cc:96] loaded metadata for tablet 1833d3ee7bc948ffb34418027b72b0ea (table default.Table_a4ca7b6127064c19b73aee408bc49282 [id=f1ff0d130d854a349d415b1d7d91a836])
I20260317 21:19:18.265362 8292 tablet_loader.cc:96] loaded metadata for tablet 18e4b69574e44eb28a735edc64e97542 (table default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065])
I20260317 21:19:18.267164 8292 tablet_loader.cc:96] loaded metadata for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8])
I20260317 21:19:18.268338 8292 tablet_loader.cc:96] loaded metadata for tablet 1ab2868e51e74828bdd3d68ddbb0477e (table default.Table_d3b3ef2c53324b2ba62c7440297657bb [id=cc8571e1fc7a4721a53d0deab165acc0])
I20260317 21:19:18.270160 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f0545b61d2d434ea4e51a8f2c4c4d3e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:18.270345 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f34026fd2f4575bb86a9dadb733fe0" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:18.271399 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fc38947219047b5abdf71af5554a987" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:18.274152 7459 tablet_bootstrap.cc:492] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:18.274705 7459 ts_tablet_manager.cc:1403] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.421s user 0.019s sys 0.014s
I20260317 21:19:18.277750 7459 raft_consensus.cc:359] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.258929 7462 consensus_queue.cc:260] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.294700 8292 tablet_loader.cc:96] loaded metadata for tablet 203f33157a0d4b82aab44da452aa46dd (table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6])
I20260317 21:19:18.296057 8292 tablet_loader.cc:96] loaded metadata for tablet 2152e5cbd2ce4159b0901ba2ffe41f40 (table default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857])
I20260317 21:19:18.297230 8292 tablet_loader.cc:96] loaded metadata for tablet 21b1c7a203ef4c22b3a84631cadfa62a (table default.Table_7fee5e45b4aa4a93900162f41e3a3d60 [id=a73a2c0e2c814324ba832290ee403c4a])
I20260317 21:19:18.298409 8292 tablet_loader.cc:96] loaded metadata for tablet 21e607674c734eaea207ed454097791d (table default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc])
I20260317 21:19:18.299669 8292 tablet_loader.cc:96] loaded metadata for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9])
I20260317 21:19:18.307320 7462 ts_tablet_manager.cc:1434] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.103s user 0.019s sys 0.000s
I20260317 21:19:18.328949 8292 tablet_loader.cc:96] loaded metadata for tablet 2a221dfad830428db2329d6e543f41dc (table default.Table_5ebd4f0e4c284b4caac2e5f0cddf4e07 [id=2d22abe029d348c484e6aff5df8b3d29])
I20260317 21:19:18.331125 8292 tablet_loader.cc:96] loaded metadata for tablet 2ccf1aa8fc9849978c752f1cebccb535 (table default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690])
I20260317 21:19:18.333118 8292 tablet_loader.cc:96] loaded metadata for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67])
I20260317 21:19:18.334347 8292 tablet_loader.cc:96] loaded metadata for tablet 3130fb84aeed42c0bba58b970150c983 (table default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee])
I20260317 21:19:18.336169 8292 tablet_loader.cc:96] loaded metadata for tablet 3180d86564c945218ec4ac5b35634047 (table default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 [id=54eed51395b74f46b69a56cfa73f3724])
I20260317 21:19:18.291114 7467 raft_consensus.cc:359] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.344363 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f0545b61d2d434ea4e51a8f2c4c4d3e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:18.359521 8292 tablet_loader.cc:96] loaded metadata for tablet 343047b78fb345729c6b66454e2cab40 (table default.Table_7864164e6f474a35a89094d36d4254ec [id=f2daa1d797d547ff9c87f31e4560581d])
I20260317 21:19:18.360517 7459 raft_consensus.cc:385] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:18.360999 7459 raft_consensus.cc:740] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:18.361908 7459 consensus_queue.cc:260] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.364866 7459 ts_tablet_manager.cc:1434] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.090s user 0.007s sys 0.000s
I20260317 21:19:18.346771 7467 raft_consensus.cc:385] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:18.372982 8495 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 7 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.373579 8495 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 7 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:18.375636 8292 tablet_loader.cc:96] loaded metadata for tablet 3d15d835328d4f7b8fea41cbedf203fe (table default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417])
I20260317 21:19:18.375732 8495 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 8 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:18.377589 8292 tablet_loader.cc:96] loaded metadata for tablet 3de1582e47f948cfa6c3c0c997ea573f (table default.Table_9967a6acfcf84b2c9759c612f25702e6 [id=790dbba0c4ed469a8d5e208209cb5f39])
I20260317 21:19:18.378767 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 8 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:18.379509 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 7 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 7.
I20260317 21:19:18.381489 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 8 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:19:18.384989 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 8 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:18.383453 8495 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 7 FOLLOWER]: Leader pre-election won for term 8
I20260317 21:19:18.385658 6683 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 5.
I20260317 21:19:18.387174 8496 raft_consensus.cc:493] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.386219 8495 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 7 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:18.387651 8496 raft_consensus.cc:515] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.389627 8496 leader_election.cc:290] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:18.388212 8495 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:19:18.380419 8292 tablet_loader.cc:96] loaded metadata for tablet 43702e625cb147339bd5871a5ee83e17 (table default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982])
I20260317 21:19:18.402998 8292 tablet_loader.cc:96] loaded metadata for tablet 445b91a6d1294c44aea0935b34be16c0 (table default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07])
I20260317 21:19:18.404399 8292 tablet_loader.cc:96] loaded metadata for tablet 4495961a559549629dedf8ce032abc9b (table default.Table_525a8435bbe14a13a70b022bb2c07f59 [id=539b531cf22a4816b0aff0f3908b0439])
I20260317 21:19:18.406858 8292 tablet_loader.cc:96] loaded metadata for tablet 4737071c2aec4449a05bead6af7ad32b (table default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552])
I20260317 21:19:18.375948 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fc38947219047b5abdf71af5554a987" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:18.380388 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f34026fd2f4575bb86a9dadb733fe0" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:18.417629 7261 leader_election.cc:343] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.421500 8292 tablet_loader.cc:96] loaded metadata for tablet 47ce1e486be64aa2803430a33c780348 (table default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e])
I20260317 21:19:18.423786 8292 tablet_loader.cc:96] loaded metadata for tablet 4c19bac7772a498f87f48e6095310347 (table default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569])
I20260317 21:19:18.425185 8292 tablet_loader.cc:96] loaded metadata for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6 (table default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82])
I20260317 21:19:18.426420 8292 tablet_loader.cc:96] loaded metadata for tablet 4f33574134934f048632aae269244368 (table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb])
I20260317 21:19:18.428112 8292 tablet_loader.cc:96] loaded metadata for tablet 516540c0439d4b75a3ac1689b8964bf7 (table default.Table_6254d435474446bca35e4c78478b186f [id=a796a06b6825482b969b97c9da0ab79d])
I20260317 21:19:18.418919 7467 raft_consensus.cc:740] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:18.441402 7467 consensus_queue.cc:260] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.444078 7467 ts_tablet_manager.cc:1434] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.264s user 0.013s sys 0.008s
I20260317 21:19:18.451140 8439 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.451720 8439 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.458226 8292 tablet_loader.cc:96] loaded metadata for tablet 54e14c247b10477fafc6ccd6a5be3761 (table default.Table_15292a1c98be431b9d3cacbc7e767c75 [id=0f32cb4e70be48c2a548e9e2d4805822])
I20260317 21:19:18.459442 8292 tablet_loader.cc:96] loaded metadata for tablet 5536ae492ac5492b8349cbadd4a156c9 (table default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 [id=b1d87190d48c47288ea093aa65a528a0])
I20260317 21:19:18.465948 8292 tablet_loader.cc:96] loaded metadata for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee])
I20260317 21:19:18.466501 8439 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:18.467583 8292 tablet_loader.cc:96] loaded metadata for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d])
I20260317 21:19:18.468971 8292 tablet_loader.cc:96] loaded metadata for tablet 57e721ac4b24438b9b2a0e2569027236 (table default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab])
I20260317 21:19:18.470296 8292 tablet_loader.cc:96] loaded metadata for tablet 585a5f0834614eb1aa4c3e405fa9bbc8 (table default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017])
I20260317 21:19:18.495555 8292 tablet_loader.cc:96] loaded metadata for tablet 5be44990b15345e1b827b8e954dd034d (table default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e])
I20260317 21:19:18.501576 8292 tablet_loader.cc:96] loaded metadata for tablet 5e080308f93448138a4846ab6bec7dfa (table default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7])
I20260317 21:19:18.504218 8292 tablet_loader.cc:96] loaded metadata for tablet 611ea189ce5d4d3c8c93c18bfd6508e7 (table default.Table_c928476e679c41589639228020653020 [id=25a841c1c6e2466eb130c7fac5dd1e68])
I20260317 21:19:18.505465 8292 tablet_loader.cc:96] loaded metadata for tablet 613e735b0f414ddf8888be0dfd23ada3 (table default.Table_147e7f28d4344faf9b6289f6c999b6a9 [id=fd0410563367415d8e73a23c52e8e839])
I20260317 21:19:18.417567 8456 consensus_queue.cc:1048] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:18.507630 8292 tablet_loader.cc:96] loaded metadata for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f])
I20260317 21:19:18.460750 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:18.508831 7327 raft_consensus.cc:2393] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
W20260317 21:19:18.519981 7264 leader_election.cc:343] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.497704 7467 tablet_bootstrap.cc:492] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:18.520440 7264 leader_election.cc:304] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:18.521524 8456 raft_consensus.cc:2749] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:18.522601 7264 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:18.523362 8456 raft_consensus.cc:2749] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260317 21:19:18.525305 7264 leader_election.cc:343] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.526322 8431 consensus_queue.cc:1048] T 8d94620882364ed3929ee314814910c9 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:18.530772 8292 tablet_loader.cc:96] loaded metadata for tablet 69801721e2b7422eba08becf2bcecfa2 (table default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0])
I20260317 21:19:18.530965 7462 tablet_bootstrap.cc:492] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:18.532793 8292 tablet_loader.cc:96] loaded metadata for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58])
I20260317 21:19:18.534051 8292 tablet_loader.cc:96] loaded metadata for tablet 6cf1a0922ed54674aaff8b0f49751c13 (table default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0])
I20260317 21:19:18.535262 8292 tablet_loader.cc:96] loaded metadata for tablet 6deece45839849088dad3d873cc925b2 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d])
I20260317 21:19:18.534560 7264 leader_election.cc:304] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:18.493913 8495 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 8 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:18.553483 8292 tablet_loader.cc:96] loaded metadata for tablet 6f980bd133da4b04b101e8df99ccca6f (table default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696])
I20260317 21:19:18.554913 8292 tablet_loader.cc:96] loaded metadata for tablet 70472ad137f34237a5a67f74c94aa36b (table default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93])
I20260317 21:19:18.557269 8292 tablet_loader.cc:96] loaded metadata for tablet 722d940aee014df4aeaddbc7aace4f4e (table default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3])
I20260317 21:19:18.558493 8292 tablet_loader.cc:96] loaded metadata for tablet 7377c42c199b408daa7f7e730ec5db06 (table default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a])
W20260317 21:19:18.554450 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 136536us
I20260317 21:19:18.566277 8456 raft_consensus.cc:2749] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:18.574952 8292 tablet_loader.cc:96] loaded metadata for tablet 74d445b420134ef5aef51f9c8e0f55da (table default.Table_cafc44dd71714a528bda6622dac7997e [id=c11557d0407d4bf08dac964b24545e44])
I20260317 21:19:18.576531 8292 tablet_loader.cc:96] loaded metadata for tablet 76bd0f5369cf400299039e4631ed11b0 (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05])
I20260317 21:19:18.577867 8292 tablet_loader.cc:96] loaded metadata for tablet 788f0ee1f7de47a49c5601c9b1494885 (table default.Table_a33d113c213c48728df41846ad097ead [id=df5c9c1007aa4e84aa9363779d23864b])
I20260317 21:19:18.580216 8292 tablet_loader.cc:96] loaded metadata for tablet 79dc6cb9627d490498950b28fbcbba8f (table default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6])
I20260317 21:19:18.581502 8292 tablet_loader.cc:96] loaded metadata for tablet 7b7b54cbff0c4fd5813d197ddbc899cc (table default.Table_d4a5487a2b7e4218b41223f155418e27 [id=b816766f31714663b685cfcc9a9f537a])
I20260317 21:19:18.582713 8292 tablet_loader.cc:96] loaded metadata for tablet 7d19b386f49a420f8561dd9c2f415aa4 (table default.Table_4827867befa741f69f6ab15938cfe436 [id=5b18e88ee5b64c1fb5f7e361c866f44a])
I20260317 21:19:18.570132 7462 tablet_bootstrap.cc:654] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:18.621230 7459 tablet_bootstrap.cc:492] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:18.622862 8292 tablet_loader.cc:96] loaded metadata for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b (table default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97])
I20260317 21:19:18.624125 8292 tablet_loader.cc:96] loaded metadata for tablet 866fdd003dd14bb9a837658e9e730f54 (table default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96])
W20260317 21:19:18.573163 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 48766us
I20260317 21:19:18.637166 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 8 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:19:18.637823 6683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 5 FOLLOWER]: Advancing to term 8
I20260317 21:19:18.640101 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 8 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
I20260317 21:19:18.640619 7918 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 7 FOLLOWER]: Advancing to term 8
I20260317 21:19:18.646598 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 8.
I20260317 21:19:18.647310 6683 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 8.
I20260317 21:19:18.647756 6737 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 8 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:19:18.649585 8292 tablet_loader.cc:96] loaded metadata for tablet 86ae75c4ce214954beecfab9275d0ea9 (table default.Table_15fe620085e9499295d76e3a7b3df064 [id=242dcdafc2b74e1ca17839f96e45af88])
I20260317 21:19:18.653270 8292 tablet_loader.cc:96] loaded metadata for tablet 8a2df0b2a73f4ed4b349efbb4d424afa (table default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a])
I20260317 21:19:18.654542 8292 tablet_loader.cc:96] loaded metadata for tablet 8a44a297079347bab6f8be6abe332bfa (table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b])
W20260317 21:19:18.660609 7261 leader_election.cc:343] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.656515 8292 tablet_loader.cc:96] loaded metadata for tablet 8c214dfdafc9464ea32fffd8223ae990 (table default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193 [id=121b3b215a0b4914a3f18167636ce46e])
I20260317 21:19:18.675330 8292 tablet_loader.cc:96] loaded metadata for tablet 8d94620882364ed3929ee314814910c9 (table default.Table_38fe3ad6f1bf4796af58d99b53c46560 [id=872cbb89b0d1405e9858d37dc22c05c4])
W20260317 21:19:18.674831 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 112748us
I20260317 21:19:18.683849 8511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 8 FOLLOWER]: Leader election won for term 8
I20260317 21:19:18.653144 6973 leader_election.cc:304] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:18.696810 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 43772.3us
I20260317 21:19:18.697454 7462 tablet_bootstrap.cc:492] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:18.698160 7462 ts_tablet_manager.cc:1403] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.167s user 0.023s sys 0.003s
I20260317 21:19:18.677465 8292 tablet_loader.cc:96] loaded metadata for tablet 932a5eb6cc2848f6a7d7c02ea47e5100 (table default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7])
I20260317 21:19:18.701032 8383 raft_consensus.cc:2804] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:18.702455 8383 raft_consensus.cc:493] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:18.702965 8383 raft_consensus.cc:3060] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:18.703078 8292 tablet_loader.cc:96] loaded metadata for tablet 9531ac1602894c43b9c165d9a5eddc76 (table default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32])
I20260317 21:19:18.703780 8456 consensus_queue.cc:1048] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:18.704308 8292 tablet_loader.cc:96] loaded metadata for tablet 983d205a22b04c8fa5ae45849fc583ab (table default.Table_db13c9846b0049dca4c784a97c434893 [id=305ca4adf2674d4c8cfa2b63e2316871])
I20260317 21:19:18.708918 8495 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 8 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:18.714154 8292 tablet_loader.cc:96] loaded metadata for tablet 99c6483618044272890b59f6b7447081 (table default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f])
I20260317 21:19:18.715528 8292 tablet_loader.cc:96] loaded metadata for tablet 99dc0fd2382142c88a80fc7eea173efc (table default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa])
I20260317 21:19:18.716835 8292 tablet_loader.cc:96] loaded metadata for tablet 9a1819f74297406e8ebb28ae847abdd3 (table default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b])
I20260317 21:19:18.718225 8292 tablet_loader.cc:96] loaded metadata for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa])
I20260317 21:19:18.701056 7462 raft_consensus.cc:359] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.721637 8456 consensus_queue.cc:1048] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:18.721799 7462 raft_consensus.cc:385] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 21:19:18.711755 7261 leader_election.cc:343] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:18.733656 7261 leader_election.cc:343] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.729125 7462 raft_consensus.cc:740] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:18.745111 8292 tablet_loader.cc:96] loaded metadata for tablet 9c3b084b04e944d7b919ec1965742e9f (table default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b])
I20260317 21:19:18.747467 8292 tablet_loader.cc:96] loaded metadata for tablet a37e53e326914575830e30915ad69cc7 (table default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71])
I20260317 21:19:18.747169 8456 consensus_queue.cc:1048] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:18.760205 7462 consensus_queue.cc:260] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.761884 8292 tablet_loader.cc:96] loaded metadata for tablet a4d64c48641b484ab443380808544d2f (table default.Table_aacec8c3d7ba49f9b6742baf37b423a8 [id=f966bc12b8824b708f7ff3d9f5789eb8])
I20260317 21:19:18.762645 7462 ts_tablet_manager.cc:1434] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.064s user 0.012s sys 0.001s
I20260317 21:19:18.763154 8292 tablet_loader.cc:96] loaded metadata for tablet a5207adefc074c229e4020f5c9406c59 (table default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b])
I20260317 21:19:18.765200 8292 tablet_loader.cc:96] loaded metadata for tablet a8a02da14e4c4e0a9ce93e60255c68a9 (table default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547])
I20260317 21:19:18.767884 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d15d835328d4f7b8fea41cbedf203fe" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:18.768244 8292 tablet_loader.cc:96] loaded metadata for tablet aa814dcc2499482eaf0afc5d95bb40f1 (table default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436])
W20260317 21:19:18.751897 7264 leader_election.cc:343] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.769372 7264 leader_election.cc:304] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:18.773069 8456 raft_consensus.cc:2749] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:18.793879 8511 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 8 LEADER]: Becoming Leader. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:18.794485 8292 tablet_loader.cc:96] loaded metadata for tablet ad8163635c344d29b8aaa2275539bd22 (table default.Table_1342223726fb480aba4932d06e045aaf [id=06e60ad198054f26b5512c8f53e70753])
I20260317 21:19:18.794981 8511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 422, Committed index: 422, Last appended: 5.422, Last appended by leader: 422, Current term: 8, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:18.796252 8292 tablet_loader.cc:96] loaded metadata for tablet ae1797cc7fc346368c11ec68dcac3b2c (table default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06])
I20260317 21:19:18.797174 8456 raft_consensus.cc:493] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.797652 8456 raft_consensus.cc:515] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.799100 8292 tablet_loader.cc:96] loaded metadata for tablet b03b54b989d04e0990ee3ccfe7a104a4 (table default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac])
I20260317 21:19:18.799571 8456 leader_election.cc:290] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:18.800359 7264 leader_election.cc:343] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.800789 7264 leader_election.cc:304] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:18.802508 8456 raft_consensus.cc:2749] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:18.805691 8456 raft_consensus.cc:493] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.806104 8456 raft_consensus.cc:515] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.807857 8456 leader_election.cc:290] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:18.820689 8292 tablet_loader.cc:96] loaded metadata for tablet b1d9d0828d4646b9952b0162a97d61a8 (table default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af])
I20260317 21:19:18.821444 7327 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "740ae685629041f39bbbc8ce715d662e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:18.824461 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:18.825076 8292 tablet_loader.cc:96] loaded metadata for tablet b35816e1de7d4b13927848538cdc3e78 (table default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a])
I20260317 21:19:18.825006 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:18.827251 8292 tablet_loader.cc:96] loaded metadata for tablet b492743dfc4d48398fbae0ee56882aa3 (table default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541])
I20260317 21:19:18.828284 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d15d835328d4f7b8fea41cbedf203fe" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:18.829147 8292 tablet_loader.cc:96] loaded metadata for tablet b516c62d56fb43f2a7c73be237cac63b (table default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072])
W20260317 21:19:18.819943 7264 leader_election.cc:343] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.830479 8292 tablet_loader.cc:96] loaded metadata for tablet b5a797b22cf34483b0061955da6b2634 (table default.Table_6157fa5cda0e4c34817e01c6bdde4904 [id=4a69de09e4be4630994811c7e432c315])
I20260317 21:19:18.839712 7459 tablet_bootstrap.cc:654] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:18.840621 8456 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.841107 8456 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.843019 8456 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:18.788621 7467 tablet_bootstrap.cc:654] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:18.848132 7264 leader_election.cc:304] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:18.854569 8456 raft_consensus.cc:2749] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:18.861454 8292 tablet_loader.cc:96] loaded metadata for tablet b68293996a25478b81d5202cb61e394a (table default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b])
I20260317 21:19:18.863708 8292 tablet_loader.cc:96] loaded metadata for tablet b76fddff23d04a9a91ea26a366249784 (table default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049])
I20260317 21:19:18.869433 8456 consensus_queue.cc:1048] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:18.874301 8431 raft_consensus.cc:493] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:18.875038 8431 raft_consensus.cc:515] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:18.866465 7261 leader_election.cc:343] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:18.880136 8431 leader_election.cc:290] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:18.881947 8292 tablet_loader.cc:96] loaded metadata for tablet b8c9cd4f760f44c88cdd986542755443 (table default.Table_24ec4c242298468dba3b539b6f272362 [id=fb617cbcbd634696a101417011a3a53a])
I20260317 21:19:18.884874 8292 tablet_loader.cc:96] loaded metadata for tablet b9ded59d55424c4990d2936f92893bbe (table default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139])
I20260317 21:19:18.886209 8292 tablet_loader.cc:96] loaded metadata for tablet bbc7e853c2474ba9bc6981601fdc820f (table default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf])
I20260317 21:19:18.893327 7462 tablet_bootstrap.cc:492] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:18.899334 7462 tablet_bootstrap.cc:654] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:18.901455 8292 tablet_loader.cc:96] loaded metadata for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2])
I20260317 21:19:18.904459 8292 tablet_loader.cc:96] loaded metadata for tablet bdea6939a3e64a279acef3b987e359a3 (table default.Table_39092d36aadb42628dabdb2bf48c1950 [id=831b6f16d49a402e8cc3d53234e050ba])
I20260317 21:19:18.908231 8292 tablet_loader.cc:96] loaded metadata for tablet c936edddda144fd7b0d6467cdc5ac5cc (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859])
I20260317 21:19:18.946628 8292 tablet_loader.cc:96] loaded metadata for tablet cbef3d2247dd4b3ba7e1a2495403d8ef (table default.Table_61b756f1959b46f393c2a726704e78aa [id=520224705f0d45feb82bb3a96db2cca0])
I20260317 21:19:18.948623 8292 tablet_loader.cc:96] loaded metadata for tablet d05973b63a8c445995ca51d8aec7bcf9 (table default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3])
W20260317 21:19:18.950369 7261 leader_election.cc:343] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:18.951020 7261 leader_election.cc:343] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:18.952533 6661 leader_election.cc:336] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: RPC error from VoteRequest() call to peer 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779): Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.557s (SENT)
I20260317 21:19:18.952977 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 6 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:18.957424 8292 tablet_loader.cc:96] loaded metadata for tablet d0769fc5106f441ba7060ed4fa681316 (table default.Table_2cd62a2d2daa4868a58114c426bca181 [id=ab3da0a6fea343d3b2d399b3437fe6f1])
I20260317 21:19:18.958868 8292 tablet_loader.cc:96] loaded metadata for tablet d2aba7eba6df430e86e003ef3a179912 (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915])
I20260317 21:19:18.961905 8292 tablet_loader.cc:96] loaded metadata for tablet d51e3d4702724484b776a95eb5acf7ef (table default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f])
I20260317 21:19:18.964156 8292 tablet_loader.cc:96] loaded metadata for tablet d69e14125d7d4412869c20759acc2336 (table default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9])
I20260317 21:19:18.968323 8495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:18.969278 8495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:18.984699 7462 tablet_bootstrap.cc:492] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:18.985133 7462 ts_tablet_manager.cc:1403] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.092s user 0.008s sys 0.007s
I20260317 21:19:18.987654 8292 tablet_loader.cc:96] loaded metadata for tablet d8247f6d37624ca3afcf0702eb6bdd55 (table default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0])
I20260317 21:19:18.988775 8292 tablet_loader.cc:96] loaded metadata for tablet d85227a2e355424d92e605dd53d9fd14 (table default.Table_9c887813658144a5ab397c605d8954c0 [id=def26fdbf280448380ad71d8c52b9713])
I20260317 21:19:18.989012 7462 raft_consensus.cc:359] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.989687 7462 raft_consensus.cc:385] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:18.989967 7462 raft_consensus.cc:740] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:18.990764 8292 tablet_loader.cc:96] loaded metadata for tablet dde40a1c77854adcabc2fff3baaf4c81 (table default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479])
I20260317 21:19:18.990561 7462 consensus_queue.cc:260] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:18.991973 8292 tablet_loader.cc:96] loaded metadata for tablet dfff4c8478b64436a666a14c133c36d3 (table default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0])
I20260317 21:19:18.992197 7462 ts_tablet_manager.cc:1434] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:19.011866 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:19.017182 7467 tablet_bootstrap.cc:492] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:19.017688 7467 ts_tablet_manager.cc:1403] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.525s user 0.016s sys 0.004s
I20260317 21:19:19.020969 8292 tablet_loader.cc:96] loaded metadata for tablet e06a1488c60f4d1387417ce30579fbf4 (table default.Table_54698fcc6a7a46deaba83055d4a1c78d [id=3e906ea0e8af47e383d0403c155be644])
I20260317 21:19:19.023363 8292 tablet_loader.cc:96] loaded metadata for tablet e1094068903840a8ab5d07c7fb0635bf (table default.Table_08bd63c86aca4345bc3ca2746d45456e [id=93623d5786f645c7a1ad496e656caa24])
I20260317 21:19:19.035342 7467 raft_consensus.cc:359] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.036123 7467 raft_consensus.cc:385] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.036422 7467 raft_consensus.cc:740] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.040331 8292 tablet_loader.cc:96] loaded metadata for tablet e22772281bd74754bb969e87a70d8077 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2])
I20260317 21:19:19.040571 7467 consensus_queue.cc:260] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.044548 7462 tablet_bootstrap.cc:492] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.049629 7467 ts_tablet_manager.cc:1434] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.032s user 0.007s sys 0.000s
I20260317 21:19:19.050562 7467 tablet_bootstrap.cc:492] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:19.061491 8292 tablet_loader.cc:96] loaded metadata for tablet e2c52f4843234e8584a66e5e8e065667 (table default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2])
I20260317 21:19:19.063002 8292 tablet_loader.cc:96] loaded metadata for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8])
I20260317 21:19:19.064316 8292 tablet_loader.cc:96] loaded metadata for tablet e586f11c37184506a15d726b4eb333be (table default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3])
I20260317 21:19:19.066459 8292 tablet_loader.cc:96] loaded metadata for tablet e988430ec1074531b0e173e96983a0cb (table default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677])
I20260317 21:19:19.069908 7467 tablet_bootstrap.cc:654] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.079847 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:19.080754 7033 raft_consensus.cc:2393] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:19.091660 7467 tablet_bootstrap.cc:492] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:19.092165 7467 ts_tablet_manager.cc:1403] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.042s user 0.009s sys 0.000s
I20260317 21:19:19.100315 8292 tablet_loader.cc:96] loaded metadata for tablet eb3a61ed542741d88d89ef6b4c53e9e6 (table default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373])
I20260317 21:19:19.095151 7467 raft_consensus.cc:359] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.101346 7467 raft_consensus.cc:385] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.101632 7467 raft_consensus.cc:740] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.101890 8292 tablet_loader.cc:96] loaded metadata for tablet ec5aa338aa2a41f0a7aeb4194189208f (table default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee])
I20260317 21:19:19.102250 7467 consensus_queue.cc:260] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.102980 8292 tablet_loader.cc:96] loaded metadata for tablet edd09d2a9ba741bc94ac682018337a83 (table default.Table_d821b351490e4553b71a09c4055e26cd [id=87c45aeb0b21447bac6301a6b5845376])
I20260317 21:19:19.107445 8383 raft_consensus.cc:515] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.114151 7467 ts_tablet_manager.cc:1434] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.022s user 0.007s sys 0.001s
I20260317 21:19:19.115288 7467 tablet_bootstrap.cc:492] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:19.118470 8431 raft_consensus.cc:493] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.119045 8431 raft_consensus.cc:515] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.123935 8431 leader_election.cc:290] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:19.126380 7467 tablet_bootstrap.cc:654] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.118937 8292 tablet_loader.cc:96] loaded metadata for tablet efd2f45acf114e3ca577103de845c5a1 (table default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c])
I20260317 21:19:19.129124 8526 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Leader pre-election lost for term 6. Reason: could not achieve majority
I20260317 21:19:19.130218 8292 tablet_loader.cc:96] loaded metadata for tablet f044d4c56cf24215a7f0beec13ece53a (table default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05])
W20260317 21:19:19.131925 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 179331us
I20260317 21:19:19.137598 7324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:19.138404 7324 raft_consensus.cc:2468] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:19.138645 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18e4b69574e44eb28a735edc64e97542" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:19.139443 7119 leader_election.cc:304] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:19.140570 8509 raft_consensus.cc:2804] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:19.140914 8509 raft_consensus.cc:493] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:19.141830 8509 raft_consensus.cc:3060] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.148942 7467 tablet_bootstrap.cc:492] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:19.149410 7467 ts_tablet_manager.cc:1403] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.034s user 0.007s sys 0.003s
I20260317 21:19:19.152307 8292 tablet_loader.cc:96] loaded metadata for tablet f5027b892229491eb5a829cacd29a700 (table default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e])
I20260317 21:19:19.152289 7467 raft_consensus.cc:359] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.153154 7467 raft_consensus.cc:385] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.153456 7467 raft_consensus.cc:740] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.153653 7462 tablet_bootstrap.cc:654] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.154089 7467 consensus_queue.cc:260] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.155973 7467 ts_tablet_manager.cc:1434] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260317 21:19:19.156955 7467 tablet_bootstrap.cc:492] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:19.162315 7467 tablet_bootstrap.cc:654] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.174111 7467 tablet_bootstrap.cc:492] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:19.174568 7467 ts_tablet_manager.cc:1403] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.018s user 0.011s sys 0.005s
I20260317 21:19:19.155972 8292 tablet_loader.cc:96] loaded metadata for tablet f69403b3b811419e87916a381656e1ec (table default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2])
I20260317 21:19:19.176895 7467 raft_consensus.cc:359] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.177579 7467 raft_consensus.cc:385] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.177968 7467 raft_consensus.cc:740] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.178587 7467 consensus_queue.cc:260] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.179445 8292 tablet_loader.cc:96] loaded metadata for tablet f6cd3f64f9b94d089b8ecf2462007ee6 (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b])
I20260317 21:19:19.180986 7467 ts_tablet_manager.cc:1434] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.006s user 0.005s sys 0.001s
I20260317 21:19:19.182587 8292 tablet_loader.cc:96] loaded metadata for tablet f7ec20308a9244a49992e378fa7b4f6c (table default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793])
I20260317 21:19:19.182700 7467 tablet_bootstrap.cc:492] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:19.197010 8509 raft_consensus.cc:515] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.185755 8292 tablet_loader.cc:96] loaded metadata for tablet f9e98c10a83e429a98a0fb4328e1b0ee (table default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48])
I20260317 21:19:19.204174 8292 tablet_loader.cc:96] loaded metadata for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f])
I20260317 21:19:19.217710 8292 tablet_loader.cc:96] loaded metadata for tablet fe51fd1fdd864b6f8b44cb45ba332c4d (table default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19])
I20260317 21:19:19.219096 8292 tablet_loader.cc:96] loaded metadata for tablet ff04afc5dc834ac4a51f97e29e4cc96e (table default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845])
W20260317 21:19:19.227130 7261 leader_election.cc:343] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.245023 7462 tablet_bootstrap.cc:492] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.245473 7462 ts_tablet_manager.cc:1403] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.201s user 0.012s sys 0.000s
I20260317 21:19:19.247834 7462 raft_consensus.cc:359] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:19.250377 8292 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Loading table and tablet metadata into memory: real 4.041s user 0.562s sys 0.103s
I20260317 21:19:19.251550 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:19.252104 7329 raft_consensus.cc:3060] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.256775 8509 leader_election.cc:290] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:19.248483 7462 raft_consensus.cc:385] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.259675 7462 raft_consensus.cc:740] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.260295 7462 consensus_queue.cc:260] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.250752 6752 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 7 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
W20260317 21:19:19.261533 6752 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 10) took 1353 ms (client timeout 904 ms). Trace:
I20260317 21:19:19.251346 6754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:19.262379 6754 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32962 (request call id 385) took 1527 ms (client timeout 557 ms). Trace:
I20260317 21:19:19.251981 6755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:19.262609 7462 ts_tablet_manager.cc:1434] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
W20260317 21:19:19.263727 6755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 8) took 2924 ms (client timeout 699 ms). Trace:
W20260317 21:19:19.264060 6755 rpcz_store.cc:269] 0317 21:19:16.339282 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:16.339519 (+ 237us) service_pool.cc:225] Handling call
0317 21:19:19.263664 (+2924145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.265267 6755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 7 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:19.252445 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:19.266358 6759 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32962 (request call id 383) took 3756 ms (client timeout 589 ms). Trace:
W20260317 21:19:19.266983 6759 rpcz_store.cc:269] 0317 21:19:15.509997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:15.510143 (+ 146us) service_pool.cc:225] Handling call
0317 21:19:19.266301 (+3756158us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.268126 6759 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 7 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:19:19.252955 6760 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:19.269464 6760 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 5) took 3682 ms (client timeout 699 ms). Trace:
W20260317 21:19:19.269811 6760 rpcz_store.cc:269] 0317 21:19:15.587278 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:15.587479 (+ 201us) service_pool.cc:225] Handling call
0317 21:19:19.269410 (+3681931us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.271155 7467 tablet_bootstrap.cc:654] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.271111 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 7 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
W20260317 21:19:19.271463 6754 rpcz_store.cc:269] 0317 21:19:17.734594 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:17.734859 (+ 265us) service_pool.cc:225] Handling call
0317 21:19:19.262316 (+1527457us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 21:19:19.272325 6752 rpcz_store.cc:269] 0317 21:19:17.908616 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:17.909253 (+ 637us) service_pool.cc:225] Handling call
0317 21:19:19.261455 (+1352202us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.253456 6753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 7 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:19.254556 6758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:19:19.255268 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:19.256278 6749 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:50358 (request call id 31) took 3618 ms. Trace:
I20260317 21:19:19.256804 6756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:19:19.257375 6757 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 6 candidate_status { last_received { term: 5 index: 422 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:19.277650 6749 rpcz_store.cc:276] 0317 21:19:15.637436 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:15.637590 (+ 154us) service_pool.cc:225] Handling call
0317 21:19:19.256017 (+3618427us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 21:19:19.279242 6757 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32962 (request call id 384) took 3147 ms (client timeout 589 ms). Trace:
W20260317 21:19:19.279793 6757 rpcz_store.cc:269] 0317 21:19:16.131785 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:16.132035 (+ 250us) service_pool.cc:225] Handling call
0317 21:19:19.279154 (+3147119us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 21:19:19.281739 6758 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 6) took 3584 ms (client timeout 585 ms). Trace:
W20260317 21:19:19.282208 6758 rpcz_store.cc:269] 0317 21:19:15.697479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:15.697687 (+ 208us) service_pool.cc:225] Handling call
0317 21:19:19.281654 (+3583967us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 21:19:19.283331 6756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 7) took 2946 ms (client timeout 585 ms). Trace:
W20260317 21:19:19.283811 6756 rpcz_store.cc:269] 0317 21:19:16.336774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:16.337015 (+ 241us) service_pool.cc:225] Handling call
0317 21:19:19.283272 (+2946257us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.284614 7462 tablet_bootstrap.cc:492] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:19.284943 6753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 9) took 1462 ms (client timeout 840 ms). Trace:
I20260317 21:19:19.285374 8543 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Leader election lost for term 7. Reason: could not achieve majority
W20260317 21:19:19.285755 6753 rpcz_store.cc:269] 0317 21:19:17.822473 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:17.822763 (+ 290us) service_pool.cc:225] Handling call
0317 21:19:19.284881 (+1462118us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.287160 8292 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:19:19.289083 6683 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Refusing update from remote peer 21186091f2c9410ea6e585c5ef2c6291: Log matching property violated. Preceding OpId in replica: term: 5 index: 422. Preceding OpId from leader: term: 8 index: 423. (index mismatch)
I20260317 21:19:19.291292 8495 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 423, Last known committed idx: 422, Time since last communication: 0.001s
W20260317 21:19:19.301182 6751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32962 (request call id 386) took 907 ms (client timeout 557 ms). Trace:
W20260317 21:19:19.301703 6751 rpcz_store.cc:269] 0317 21:19:18.394259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:18.394489 (+ 230us) service_pool.cc:225] Handling call
0317 21:19:19.301113 (+906624us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:19.302292 7918 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Refusing update from remote peer 21186091f2c9410ea6e585c5ef2c6291: Log matching property violated. Preceding OpId in replica: term: 5 index: 422. Preceding OpId from leader: term: 8 index: 423. (index mismatch)
I20260317 21:19:19.303746 8495 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 423, Last known committed idx: 419, Time since last communication: 0.000s
I20260317 21:19:19.307498 7179 raft_consensus.cc:3060] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.308266 7462 tablet_bootstrap.cc:654] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.310571 8526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.311547 8526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:19.325569 7462 tablet_bootstrap.cc:492] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.325994 7462 ts_tablet_manager.cc:1403] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.042s user 0.001s sys 0.010s
I20260317 21:19:19.327304 7184 raft_consensus.cc:3060] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.329698 8543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.330418 8543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:19.331271 8549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.334649 8549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:19.333863 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.335798 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:19.338701 8550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.339948 8550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:19.347213 7462 raft_consensus.cc:359] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.355077 7462 raft_consensus.cc:385] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.355880 7462 raft_consensus.cc:740] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.357131 7462 consensus_queue.cc:260] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.357671 8495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.358377 8495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:19.358989 8495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.359615 8495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:19.361171 7462 ts_tablet_manager.cc:1434] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.035s user 0.004s sys 0.004s
I20260317 21:19:19.370030 7462 tablet_bootstrap.cc:492] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.370759 7329 raft_consensus.cc:2468] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:19.378083 7119 leader_election.cc:304] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:19.152117 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:19.378830 8509 raft_consensus.cc:2804] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:19.186134 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "18e4b69574e44eb28a735edc64e97542" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:19.186848 6973 leader_election.cc:304] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:19.379228 8509 raft_consensus.cc:697] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
W20260317 21:19:19.379668 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 192886us
I20260317 21:19:19.380275 8435 raft_consensus.cc:2749] T 740ae685629041f39bbbc8ce715d662e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:19.379981 8509 consensus_queue.cc:237] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.187700 7459 tablet_bootstrap.cc:492] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:19.381242 7459 ts_tablet_manager.cc:1403] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.761s user 0.014s sys 0.000s
I20260317 21:19:19.382858 8292 catalog_manager.cc:1269] Loaded cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:19:19.383193 8292 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:19:19.383611 7459 raft_consensus.cc:359] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.384251 7459 raft_consensus.cc:385] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.384548 7459 raft_consensus.cc:740] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.384892 7037 raft_consensus.cc:2393] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:19.385404 7459 consensus_queue.cc:260] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.388798 7459 ts_tablet_manager.cc:1434] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:19.395244 7462 tablet_bootstrap.cc:654] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.401968 7459 tablet_bootstrap.cc:492] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:19.408603 8431 raft_consensus.cc:493] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.409144 8431 raft_consensus.cc:515] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.411058 8431 leader_election.cc:290] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:19.422031 7462 tablet_bootstrap.cc:492] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.422468 7462 ts_tablet_manager.cc:1403] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.053s user 0.009s sys 0.002s
W20260317 21:19:19.426666 7264 leader_election.cc:343] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.427059 7264 leader_election.cc:304] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:19.428793 8456 raft_consensus.cc:2749] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:19.425856 7462 raft_consensus.cc:359] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.433766 7120 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:19.434046 7462 raft_consensus.cc:385] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.434316 7462 raft_consensus.cc:740] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.435660 7462 consensus_queue.cc:260] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.437255 7467 tablet_bootstrap.cc:492] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:19.437433 8431 raft_consensus.cc:3060] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.440219 7467 ts_tablet_manager.cc:1403] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.255s user 0.010s sys 0.015s
I20260317 21:19:19.442785 7462 ts_tablet_manager.cc:1434] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.020s user 0.006s sys 0.001s
I20260317 21:19:19.443655 7462 tablet_bootstrap.cc:492] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.443447 8517 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.443977 8517 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.446005 8517 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:19.448421 7459 tablet_bootstrap.cc:654] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.461896 7462 tablet_bootstrap.cc:654] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.464577 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:19.474608 6971 leader_election.cc:343] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.478410 7462 tablet_bootstrap.cc:492] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.478849 7462 ts_tablet_manager.cc:1403] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.001s
I20260317 21:19:19.481493 7462 raft_consensus.cc:359] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.482178 7462 raft_consensus.cc:385] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.482426 7462 raft_consensus.cc:740] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.483043 7462 consensus_queue.cc:260] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.488030 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:19.489208 7462 ts_tablet_manager.cc:1434] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.010s user 0.007s sys 0.001s
I20260317 21:19:19.490162 7462 tablet_bootstrap.cc:492] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.490456 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:19.493103 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:19.493736 7033 raft_consensus.cc:2468] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:19.497601 8292 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:19:19.499241 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4cee7952cf5428b98340789a48d2da2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:19.512804 7462 tablet_bootstrap.cc:654] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.515092 8383 leader_election.cc:290] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:19.518046 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:19.523835 7325 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:19.524488 7325 raft_consensus.cc:2468] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:19.483776 7467 raft_consensus.cc:359] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:19.525384 7264 leader_election.cc:343] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.525455 7467 raft_consensus.cc:385] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.525755 7467 raft_consensus.cc:740] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.525795 7264 leader_election.cc:304] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:19.526924 7264 leader_election.cc:343] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.526399 7467 consensus_queue.cc:260] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.527294 7264 leader_election.cc:304] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:19.527307 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 92055.9us
I20260317 21:19:19.528259 7264 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:19.529860 8533 raft_consensus.cc:2749] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:19.530464 8533 raft_consensus.cc:2749] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:19.532339 7324 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:19.533041 7324 raft_consensus.cc:2393] T 722d940aee014df4aeaddbc7aace4f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
I20260317 21:19:19.534264 8496 raft_consensus.cc:2804] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:19.534644 8496 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:19.534958 8496 raft_consensus.cc:3060] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.538367 8526 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 8 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:19.539094 8526 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:19.545254 7467 ts_tablet_manager.cc:1434] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.105s user 0.003s sys 0.004s
I20260317 21:19:19.546203 7467 tablet_bootstrap.cc:492] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:19.560192 8292 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Loaded TSK: 0
I20260317 21:19:19.562781 8496 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.564934 8292 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:19:19.567921 7462 tablet_bootstrap.cc:492] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.568423 7462 ts_tablet_manager.cc:1403] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.011s
I20260317 21:19:19.569700 8496 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:19.571681 7462 raft_consensus.cc:359] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.575817 7462 raft_consensus.cc:385] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.576289 7462 raft_consensus.cc:740] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.577065 7462 consensus_queue.cc:260] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.585325 7462 ts_tablet_manager.cc:1434] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260317 21:19:19.587791 7462 tablet_bootstrap.cc:492] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:19.598408 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 8a44a297079347bab6f8be6abe332bfa (table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8a44a297079347bab6f8be6abe332bfa consensus state has no leader
I20260317 21:19:19.602313 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4cee7952cf5428b98340789a48d2da2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:19.601099 8560 log.cc:927] Time spent T b2dccb915fad4666bd7978f8c416db44 P d3cb3b49e3e94537a6e434ea4e037c3f: Append to log took a long time: real 0.082s user 0.001s sys 0.000s
I20260317 21:19:19.608467 8568 raft_consensus.cc:493] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.609000 8568 raft_consensus.cc:515] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.611006 8568 leader_election.cc:290] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:19.621949 6973 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:19.606532 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "722d940aee014df4aeaddbc7aace4f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:19.623402 7033 raft_consensus.cc:2393] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:19.636590 7462 tablet_bootstrap.cc:654] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.637521 7459 tablet_bootstrap.cc:492] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:19.638036 7459 ts_tablet_manager.cc:1403] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.236s user 0.012s sys 0.005s
I20260317 21:19:19.635775 8292 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:19:19.653085 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:19.654865 7261 leader_election.cc:343] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.655054 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5207adefc074c229e4020f5c9406c59" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:19.656205 7261 leader_election.cc:343] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.657284 7462 tablet_bootstrap.cc:492] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.657712 7462 ts_tablet_manager.cc:1403] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.070s user 0.008s sys 0.007s
I20260317 21:19:19.660053 7462 raft_consensus.cc:359] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.660722 7462 raft_consensus.cc:385] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260317 21:19:19.662791 7091 tablet.cc:2379] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:19:19.660972 7462 raft_consensus.cc:740] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.668041 7462 consensus_queue.cc:260] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.671180 7462 ts_tablet_manager.cc:1434] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260317 21:19:19.649140 8431 raft_consensus.cc:2749] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:19.677904 7462 tablet_bootstrap.cc:492] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.680941 7467 tablet_bootstrap.cc:654] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.683399 7462 tablet_bootstrap.cc:654] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.664335 7459 raft_consensus.cc:359] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.684325 7459 raft_consensus.cc:385] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.688699 7459 raft_consensus.cc:740] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.689553 7459 consensus_queue.cc:260] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.691335 7459 ts_tablet_manager.cc:1434] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.053s user 0.003s sys 0.000s
I20260317 21:19:19.692873 7459 tablet_bootstrap.cc:492] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:19.695618 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 167475us
I20260317 21:19:19.700708 7462 tablet_bootstrap.cc:492] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.701154 7462 ts_tablet_manager.cc:1403] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.005s
I20260317 21:19:19.704396 7462 raft_consensus.cc:359] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.705163 7462 raft_consensus.cc:385] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.702016 7120 leader_election.cc:304] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:19.705444 7462 raft_consensus.cc:740] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.709239 7462 consensus_queue.cc:260] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.710681 8542 raft_consensus.cc:2749] T 18e4b69574e44eb28a735edc64e97542 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:19.718518 7462 ts_tablet_manager.cc:1434] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
W20260317 21:19:19.738229 7379 tablet.cc:2379] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260317 21:19:19.749038 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 196101us
I20260317 21:19:19.752050 7462 tablet_bootstrap.cc:492] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.758375 7462 tablet_bootstrap.cc:654] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.768787 8568 raft_consensus.cc:493] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.769515 8509 raft_consensus.cc:493] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader d3cb3b49e3e94537a6e434ea4e037c3f)
I20260317 21:19:19.769320 8568 raft_consensus.cc:515] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.770004 8509 raft_consensus.cc:515] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.772998 8568 leader_election.cc:290] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:19.774197 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70560dde990843e0b7c74ae3206c58e4" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:19.781090 7261 leader_election.cc:343] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.782486 8509 leader_election.cc:290] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:19.784277 7462 tablet_bootstrap.cc:492] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.784777 7462 ts_tablet_manager.cc:1403] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.033s user 0.010s sys 0.004s
I20260317 21:19:19.787145 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:19.789052 7462 raft_consensus.cc:359] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.789784 7462 raft_consensus.cc:385] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.790037 7462 raft_consensus.cc:740] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.796708 7462 consensus_queue.cc:260] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.799031 7462 ts_tablet_manager.cc:1434] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.014s user 0.002s sys 0.004s
I20260317 21:19:19.800091 7462 tablet_bootstrap.cc:492] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.814509 7462 tablet_bootstrap.cc:654] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.818631 7467 tablet_bootstrap.cc:492] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:19.819021 7467 ts_tablet_manager.cc:1403] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.273s user 0.004s sys 0.006s
I20260317 21:19:19.822036 7467 raft_consensus.cc:359] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.822688 7467 raft_consensus.cc:385] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.822947 7467 raft_consensus.cc:740] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.823553 7467 consensus_queue.cc:260] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.837261 8550 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:19.837867 8550 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:19.839942 8550 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:19.842291 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:19.844611 7462 tablet_bootstrap.cc:492] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.845122 7462 ts_tablet_manager.cc:1403] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.045s user 0.010s sys 0.005s
I20260317 21:19:19.851608 7462 raft_consensus.cc:359] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.857993 7462 raft_consensus.cc:385] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.858419 7462 raft_consensus.cc:740] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.862352 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9]
I20260317 21:19:19.865787 7462 consensus_queue.cc:260] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.866914 7467 ts_tablet_manager.cc:1434] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.048s user 0.005s sys 0.000s
I20260317 21:19:19.867993 7179 raft_consensus.cc:3060] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:19.869124 7462 ts_tablet_manager.cc:1434] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.020s user 0.002s sys 0.005s
I20260317 21:19:19.874068 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e]
I20260317 21:19:19.874986 7462 tablet_bootstrap.cc:492] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.881588 7462 tablet_bootstrap.cc:654] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.881577 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1342223726fb480aba4932d06e045aaf [id=06e60ad198054f26b5512c8f53e70753]
I20260317 21:19:19.887257 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017]
I20260317 21:19:19.894464 8568 raft_consensus.cc:493] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.893585 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa]
I20260317 21:19:19.895258 8568 raft_consensus.cc:515] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.897229 8568 leader_election.cc:290] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:19.897810 8570 raft_consensus.cc:493] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:19.899348 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:19.898371 8570 raft_consensus.cc:515] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.902695 8570 leader_election.cc:290] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:19.904153 7261 leader_election.cc:343] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:19.904201 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "252f728d183f440a9ef42894508d5755" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:19.904681 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7]
I20260317 21:19:19.910989 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_15292a1c98be431b9d3cacbc7e767c75 [id=0f32cb4e70be48c2a548e9e2d4805822]
I20260317 21:19:19.920735 7462 tablet_bootstrap.cc:492] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:19.920022 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8]
I20260317 21:19:19.921200 7462 ts_tablet_manager.cc:1403] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.000s
I20260317 21:19:19.923933 7462 raft_consensus.cc:359] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.924659 7462 raft_consensus.cc:385] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:19.924918 7462 raft_consensus.cc:740] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:19.926030 7462 consensus_queue.cc:260] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:19.926990 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac]
I20260317 21:19:19.938234 7462 ts_tablet_manager.cc:1434] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.017s user 0.006s sys 0.000s
I20260317 21:19:19.944264 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b]
I20260317 21:19:19.954120 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193 [id=121b3b215a0b4914a3f18167636ce46e]
I20260317 21:19:19.962076 7459 tablet_bootstrap.cc:654] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:19.972097 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373]
I20260317 21:19:19.981518 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9]
I20260317 21:19:19.983254 7462 tablet_bootstrap.cc:492] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:19.996019 7462 tablet_bootstrap.cc:654] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.001842 8517 raft_consensus.cc:2804] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:20.002569 8517 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:20.003000 8517 raft_consensus.cc:3060] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.013420 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05]
I20260317 21:19:20.023787 8568 raft_consensus.cc:493] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.024329 8568 raft_consensus.cc:515] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.029320 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.029768 8568 leader_election.cc:290] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:20.030762 7261 leader_election.cc:343] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.031342 8570 raft_consensus.cc:493] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.031880 8570 raft_consensus.cc:515] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.033388 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06]
I20260317 21:19:20.036541 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:20.044548 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19]
I20260317 21:19:20.056845 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f]
I20260317 21:19:20.064085 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7]
W20260317 21:19:20.051504 7239 tablet.cc:2379] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:19:20.040721 8570 leader_election.cc:290] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:20.074329 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]
I20260317 21:19:20.077721 7467 tablet_bootstrap.cc:492] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:20.060292 7183 raft_consensus.cc:3060] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.084919 8568 raft_consensus.cc:493] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.085486 8568 raft_consensus.cc:515] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.084230 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee]
I20260317 21:19:20.087832 8568 leader_election.cc:290] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:20.095202 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_15fe620085e9499295d76e3a7b3df064 [id=242dcdafc2b74e1ca17839f96e45af88]
I20260317 21:19:20.099200 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:20.100476 7261 leader_election.cc:343] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.105113 7467 tablet_bootstrap.cc:654] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.110965 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_c928476e679c41589639228020653020 [id=25a841c1c6e2466eb130c7fac5dd1e68]
I20260317 21:19:20.116616 7462 tablet_bootstrap.cc:492] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.117126 7462 ts_tablet_manager.cc:1403] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.134s user 0.049s sys 0.024s
I20260317 21:19:20.120977 7462 raft_consensus.cc:359] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.121688 7462 raft_consensus.cc:385] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.125428 8568 raft_consensus.cc:493] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.126116 8568 raft_consensus.cc:515] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.126842 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5ebd4f0e4c284b4caac2e5f0cddf4e07 [id=2d22abe029d348c484e6aff5df8b3d29]
I20260317 21:19:20.128182 8568 leader_election.cc:290] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.121978 7462 raft_consensus.cc:740] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.129868 7462 consensus_queue.cc:260] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.130244 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3beb9232d74e45ab713ae8241eec4c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:20.132488 7261 leader_election.cc:343] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.133867 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b]
I20260317 21:19:20.135452 7462 ts_tablet_manager.cc:1434] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.018s user 0.007s sys 0.000s
I20260317 21:19:20.142413 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc]
I20260317 21:19:20.143038 8568 raft_consensus.cc:493] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.143707 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_db13c9846b0049dca4c784a97c434893 [id=305ca4adf2674d4c8cfa2b63e2316871]
I20260317 21:19:20.143499 8568 raft_consensus.cc:515] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.144492 7467 tablet_bootstrap.cc:492] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:20.144908 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0]
I20260317 21:19:20.146096 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a]
I20260317 21:19:20.146653 8568 leader_election.cc:290] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.162181 7467 ts_tablet_manager.cc:1403] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.085s user 0.010s sys 0.002s
I20260317 21:19:20.165040 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb]
I20260317 21:19:20.166420 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479]
I20260317 21:19:20.167663 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67]
I20260317 21:19:20.168920 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_54698fcc6a7a46deaba83055d4a1c78d [id=3e906ea0e8af47e383d0403c155be644]
I20260317 21:19:20.171574 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6]
I20260317 21:19:20.174103 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_6157fa5cda0e4c34817e01c6bdde4904 [id=4a69de09e4be4630994811c7e432c315]
I20260317 21:19:20.065083 8517 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.175526 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677]
I20260317 21:19:20.176891 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d]
I20260317 21:19:20.185501 7462 tablet_bootstrap.cc:492] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:20.186244 8517 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:20.146894 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f427b15a7b2a47e18271e8884c637f4e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.199400 7462 tablet_bootstrap.cc:654] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.202790 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2]
W20260317 21:19:20.203346 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 581348us
I20260317 21:19:20.204085 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_61b756f1959b46f393c2a726704e78aa [id=520224705f0d45feb82bb3a96db2cca0]
I20260317 21:19:20.205371 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_525a8435bbe14a13a70b022bb2c07f59 [id=539b531cf22a4816b0aff0f3908b0439]
I20260317 21:19:20.207104 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 [id=54eed51395b74f46b69a56cfa73f3724]
I20260317 21:19:20.209245 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072]
I20260317 21:19:20.211185 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8]
I20260317 21:19:20.224936 7462 tablet_bootstrap.cc:492] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.225348 7462 ts_tablet_manager.cc:1403] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.040s user 0.006s sys 0.004s
I20260317 21:19:20.227427 7459 tablet_bootstrap.cc:492] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:20.228091 7459 ts_tablet_manager.cc:1403] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.535s user 0.021s sys 0.007s
I20260317 21:19:20.165134 7467 raft_consensus.cc:359] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.225726 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:20.226758 7182 raft_consensus.cc:3060] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.231648 7467 raft_consensus.cc:385] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.231957 7467 raft_consensus.cc:740] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.232761 7467 consensus_queue.cc:260] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.234916 7467 ts_tablet_manager.cc:1434] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.072s user 0.004s sys 0.000s
I20260317 21:19:20.235356 6973 leader_election.cc:304] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:20.228600 7462 raft_consensus.cc:359] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.236390 7462 raft_consensus.cc:385] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.236697 7462 raft_consensus.cc:740] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.237370 7462 consensus_queue.cc:260] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.239961 7462 ts_tablet_manager.cc:1434] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260317 21:19:20.240057 7120 leader_election.cc:304] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.240908 7462 tablet_bootstrap.cc:492] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:20.239203 8517 raft_consensus.cc:2749] T 722d940aee014df4aeaddbc7aace4f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.242059 8509 raft_consensus.cc:2749] T a4cee7952cf5428b98340789a48d2da2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.243717 7261 leader_election.cc:343] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.244941 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6]
I20260317 21:19:20.246284 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_4827867befa741f69f6ab15938cfe436 [id=5b18e88ee5b64c1fb5f7e361c866f44a]
I20260317 21:19:20.247587 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9]
I20260317 21:19:20.248834 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547]
I20260317 21:19:20.264765 7462 tablet_bootstrap.cc:654] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.266979 8542 raft_consensus.cc:493] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2f539d1a4a584b8ca83928be49369bf4)
I20260317 21:19:20.267473 8542 raft_consensus.cc:515] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.269783 8542 leader_election.cc:290] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:20.273085 8581 raft_consensus.cc:493] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.273998 8568 raft_consensus.cc:493] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.273681 8581 raft_consensus.cc:515] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.274477 8568 raft_consensus.cc:515] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.275872 8581 leader_election.cc:290] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:20.276683 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:20.279266 6971 leader_election.cc:343] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.230887 7459 raft_consensus.cc:359] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.281606 7459 raft_consensus.cc:385] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.281922 7459 raft_consensus.cc:740] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.282570 7459 consensus_queue.cc:260] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.285599 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.287320 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d15d835328d4f7b8fea41cbedf203fe" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.276428 8568 leader_election.cc:290] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.291630 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:20.292405 7322 raft_consensus.cc:2393] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
W20260317 21:19:20.297024 7261 leader_election.cc:343] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:20.300822 6971 leader_election.cc:343] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.301182 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793]
I20260317 21:19:20.303829 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611]
I20260317 21:19:20.307888 8533 raft_consensus.cc:493] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.308382 8533 raft_consensus.cc:515] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.310127 7467 tablet_bootstrap.cc:492] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:20.310346 8533 leader_election.cc:290] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.312889 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e]
I20260317 21:19:20.318919 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f]
I20260317 21:19:20.324205 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2607b9978c32466eb6dc8b8acfb429ce" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.324121 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e06a1488c60f4d1387417ce30579fbf4" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:20.325492 7261 leader_election.cc:343] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.330145 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "252f728d183f440a9ef42894508d5755" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.331758 7459 ts_tablet_manager.cc:1434] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.103s user 0.003s sys 0.004s
I20260317 21:19:20.333030 7459 tablet_bootstrap.cc:492] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:20.337930 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93]
I20260317 21:19:20.339478 8570 raft_consensus.cc:493] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.340271 8570 raft_consensus.cc:515] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.342612 8570 leader_election.cc:290] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:20.349670 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05]
I20260317 21:19:20.351075 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]
I20260317 21:19:20.359973 7467 tablet_bootstrap.cc:654] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.367053 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "203f33157a0d4b82aab44da452aa46dd" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:20.368777 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 46138.6us
I20260317 21:19:20.375947 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9967a6acfcf84b2c9759c612f25702e6 [id=790dbba0c4ed469a8d5e208209cb5f39]
I20260317 21:19:20.376291 7183 raft_consensus.cc:3060] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.377233 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0]
I20260317 21:19:20.378885 8593 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
W20260317 21:19:20.383266 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 104196us
I20260317 21:19:20.385457 8593 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:20.394004 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:20.397795 8570 consensus_queue.cc:1048] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:20.398370 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48]
I20260317 21:19:20.407516 8568 raft_consensus.cc:493] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.407991 8568 raft_consensus.cc:515] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.409744 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]
I20260317 21:19:20.409902 8568 leader_election.cc:290] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.410516 8593 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:20.411151 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:20.412405 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.413587 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_39092d36aadb42628dabdb2bf48c1950 [id=831b6f16d49a402e8cc3d53234e050ba]
W20260317 21:19:20.414048 7261 leader_election.cc:343] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.414911 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994]
I20260317 21:19:20.415709 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2a221dfad830428db2329d6e543f41dc" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.416914 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_38fe3ad6f1bf4796af58d99b53c46560 [id=872cbb89b0d1405e9858d37dc22c05c4]
I20260317 21:19:20.417665 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e586f11c37184506a15d726b4eb333be" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.418141 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d821b351490e4553b71a09c4055e26cd [id=87c45aeb0b21447bac6301a6b5845376]
I20260317 21:19:20.426261 8568 raft_consensus.cc:493] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.426476 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3]
I20260317 21:19:20.426923 8568 raft_consensus.cc:515] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.429752 7462 tablet_bootstrap.cc:492] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.429955 8568 leader_election.cc:290] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.430212 7462 ts_tablet_manager.cc:1403] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.189s user 0.017s sys 0.005s
I20260317 21:19:20.434369 7462 raft_consensus.cc:359] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.435155 7462 raft_consensus.cc:385] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.435554 7462 raft_consensus.cc:740] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.436276 7462 consensus_queue.cc:260] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.437932 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc7e853c2474ba9bc6981601fdc820f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.438479 7462 ts_tablet_manager.cc:1434] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260317 21:19:20.439502 7462 tablet_bootstrap.cc:492] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:20.432107 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845]
I20260317 21:19:20.442296 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097]
I20260317 21:19:20.443650 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982]
W20260317 21:19:20.445011 7261 leader_election.cc:343] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.446274 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3]
I20260317 21:19:20.447480 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552]
I20260317 21:19:20.460539 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:20.460603 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5207adefc074c229e4020f5c9406c59" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.461267 7462 tablet_bootstrap.cc:654] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.461536 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70560dde990843e0b7c74ae3206c58e4" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.461956 7037 raft_consensus.cc:2393] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f539d1a4a584b8ca83928be49369bf4 in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:20.464843 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_08bd63c86aca4345bc3ca2746d45456e [id=93623d5786f645c7a1ad496e656caa24]
I20260317 21:19:20.467314 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]
I20260317 21:19:20.468758 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96]
I20260317 21:19:20.470387 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0]
I20260317 21:19:20.471633 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0]
I20260317 21:19:20.487438 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]
I20260317 21:19:20.500329 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7fee5e45b4aa4a93900162f41e3a3d60 [id=a73a2c0e2c814324ba832290ee403c4a]
I20260317 21:19:20.501885 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_6254d435474446bca35e4c78478b186f [id=a796a06b6825482b969b97c9da0ab79d]
I20260317 21:19:20.513855 7462 tablet_bootstrap.cc:492] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.514338 7462 ts_tablet_manager.cc:1403] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.075s user 0.012s sys 0.004s
I20260317 21:19:20.516749 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_2cd62a2d2daa4868a58114c426bca181 [id=ab3da0a6fea343d3b2d399b3437fe6f1]
I20260317 21:19:20.516971 7462 raft_consensus.cc:359] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.517602 7462 raft_consensus.cc:385] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.517861 7462 raft_consensus.cc:740] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.518078 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139]
I20260317 21:19:20.519330 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e]
I20260317 21:19:20.518471 7462 consensus_queue.cc:260] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.521993 7462 ts_tablet_manager.cc:1434] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.007s user 0.006s sys 0.001s
I20260317 21:19:20.522892 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541]
I20260317 21:19:20.524148 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 [id=b1d87190d48c47288ea093aa65a528a0]
I20260317 21:19:20.533669 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343]
I20260317 21:19:20.534503 7467 tablet_bootstrap.cc:492] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:20.534961 7467 ts_tablet_manager.cc:1403] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.225s user 0.004s sys 0.007s
I20260317 21:19:20.535024 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af]
I20260317 21:19:20.536505 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436]
I20260317 21:19:20.538241 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b]
I20260317 21:19:20.538763 7467 raft_consensus.cc:359] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.539398 7467 raft_consensus.cc:385] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.539664 7467 raft_consensus.cc:740] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.540378 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d4a5487a2b7e4218b41223f155418e27 [id=b816766f31714663b685cfcc9a9f537a]
I20260317 21:19:20.540380 7467 consensus_queue.cc:260] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.541671 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7]
I20260317 21:19:20.543498 7467 ts_tablet_manager.cc:1434] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.008s user 0.005s sys 0.000s
I20260317 21:19:20.550511 8591 raft_consensus.cc:493] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.551000 8591 raft_consensus.cc:515] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.555315 8591 leader_election.cc:290] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.557372 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfff4c8478b64436a666a14c133c36d3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:20.558619 7261 leader_election.cc:343] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.562865 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce]
I20260317 21:19:20.564342 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f]
I20260317 21:19:20.567970 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_cafc44dd71714a528bda6622dac7997e [id=c11557d0407d4bf08dac964b24545e44]
I20260317 21:19:20.570878 7032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.572199 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3beb9232d74e45ab713ae8241eec4c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.573037 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.574121 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.587733 7032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f427b15a7b2a47e18271e8884c637f4e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.591116 7459 tablet_bootstrap.cc:654] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:20.600757 6659 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 8d94620882364ed3929ee314814910c9 (table default.Table_38fe3ad6f1bf4796af58d99b53c46560 [id=872cbb89b0d1405e9858d37dc22c05c4], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 8d94620882364ed3929ee314814910c9 consensus state has no leader
I20260317 21:19:20.602389 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.608335 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2607b9978c32466eb6dc8b8acfb429ce" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.621326 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690]
I20260317 21:19:20.636873 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07]
I20260317 21:19:20.642282 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.656343 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc7e853c2474ba9bc6981601fdc820f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.657356 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2]
I20260317 21:19:20.666590 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfff4c8478b64436a666a14c133c36d3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:20.669433 7462 tablet_bootstrap.cc:492] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:20.680136 7462 tablet_bootstrap.cc:654] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.691991 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32]
I20260317 21:19:20.699219 7462 tablet_bootstrap.cc:492] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.699682 7462 ts_tablet_manager.cc:1403] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.005s
I20260317 21:19:20.702164 7462 raft_consensus.cc:359] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.702859 7462 raft_consensus.cc:385] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.703115 7462 raft_consensus.cc:740] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.703738 7462 consensus_queue.cc:260] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.707228 7459 tablet_bootstrap.cc:492] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:20.722690 7120 leader_election.cc:304] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.724869 8570 raft_consensus.cc:2749] T e06a1488c60f4d1387417ce30579fbf4 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:20.725421 7467 tablet_bootstrap.cc:492] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:20.730263 7462 ts_tablet_manager.cc:1434] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.030s user 0.007s sys 0.000s
I20260317 21:19:20.731236 7462 tablet_bootstrap.cc:492] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:20.731974 7467 tablet_bootstrap.cc:654] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.733973 7323 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d15d835328d4f7b8fea41cbedf203fe" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:20.734835 7323 raft_consensus.cc:2468] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:20.749024 7462 tablet_bootstrap.cc:654] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.748952 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696]
I20260317 21:19:20.750461 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857]
I20260317 21:19:20.767212 7467 tablet_bootstrap.cc:492] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:20.767642 7467 ts_tablet_manager.cc:1403] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.042s user 0.005s sys 0.005s
I20260317 21:19:20.769562 7459 ts_tablet_manager.cc:1403] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.437s user 0.008s sys 0.003s
I20260317 21:19:20.770681 7467 raft_consensus.cc:359] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.771303 7467 raft_consensus.cc:385] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.771610 7467 raft_consensus.cc:740] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.775420 7462 tablet_bootstrap.cc:492] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.775849 7462 ts_tablet_manager.cc:1403] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.045s user 0.005s sys 0.004s
I20260317 21:19:20.776180 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b]
I20260317 21:19:20.772279 7467 consensus_queue.cc:260] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.784941 7467 ts_tablet_manager.cc:1434] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.017s user 0.009s sys 0.000s
I20260317 21:19:20.789809 7462 raft_consensus.cc:359] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.790541 7462 raft_consensus.cc:385] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.790818 7462 raft_consensus.cc:740] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.791709 7462 consensus_queue.cc:260] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.795171 7120 leader_election.cc:304] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.796481 8570 raft_consensus.cc:2749] T 252f728d183f440a9ef42894508d5755 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.799145 7467 tablet_bootstrap.cc:492] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:20.806195 7467 tablet_bootstrap.cc:654] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:20.813880 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82]
I20260317 21:19:20.814888 7120 leader_election.cc:304] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.818362 8509 raft_consensus.cc:2749] T 203f33157a0d4b82aab44da452aa46dd P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.794402 7462 ts_tablet_manager.cc:1434] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.018s user 0.011s sys 0.001s
I20260317 21:19:20.830369 7120 leader_election.cc:304] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.831666 8570 raft_consensus.cc:2749] T 2a221dfad830428db2329d6e543f41dc P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:20.833374 7462 tablet_bootstrap.cc:492] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:20.834877 7467 tablet_bootstrap.cc:492] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:20.835322 7467 ts_tablet_manager.cc:1403] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.036s user 0.007s sys 0.003s
I20260317 21:19:20.837845 7467 raft_consensus.cc:359] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.838519 7467 raft_consensus.cc:385] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.838785 7467 raft_consensus.cc:740] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.839380 7467 consensus_queue.cc:260] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.839332 7459 raft_consensus.cc:359] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.841747 7467 ts_tablet_manager.cc:1434] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.006s user 0.008s sys 0.000s
I20260317 21:19:20.849855 7120 leader_election.cc:304] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.851589 8509 raft_consensus.cc:2749] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.861310 7264 leader_election.cc:343] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.861800 7264 leader_election.cc:304] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.862473 8568 raft_consensus.cc:2749] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.863296 7264 leader_election.cc:343] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.863669 7264 leader_election.cc:304] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.865612 8568 raft_consensus.cc:2749] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.865767 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa]
I20260317 21:19:20.864748 7264 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.867044 8591 raft_consensus.cc:2749] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.867177 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b]
W20260317 21:19:20.868824 7264 leader_election.cc:343] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.869232 7264 leader_election.cc:304] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.869911 8591 raft_consensus.cc:2749] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.871177 7264 leader_election.cc:343] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.871531 7264 leader_election.cc:304] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.872195 8591 raft_consensus.cc:2749] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.873178 7264 leader_election.cc:343] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.873551 7264 leader_election.cc:304] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.874202 8591 raft_consensus.cc:2749] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.877825 7264 leader_election.cc:343] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.878273 7264 leader_election.cc:304] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.883045 7459 raft_consensus.cc:385] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.886188 8568 raft_consensus.cc:2749] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.886785 7459 raft_consensus.cc:740] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:20.887467 7264 leader_election.cc:343] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.887858 7264 leader_election.cc:304] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.888711 7467 tablet_bootstrap.cc:492] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:20.888847 8568 raft_consensus.cc:2749] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.889540 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065]
I20260317 21:19:20.891558 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2]
W20260317 21:19:20.892048 7264 leader_election.cc:343] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.891556 7459 consensus_queue.cc:260] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.892443 7264 leader_election.cc:304] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.893126 8568 raft_consensus.cc:2749] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.896003 7467 tablet_bootstrap.cc:654] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:20.895476 7264 leader_election.cc:343] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.896330 7264 leader_election.cc:304] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.900828 8568 raft_consensus.cc:2749] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.902163 7264 leader_election.cc:343] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.902621 7264 leader_election.cc:304] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.903348 8592 raft_consensus.cc:2749] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:20.903595 7821 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 217489us
I20260317 21:19:20.910341 7462 tablet_bootstrap.cc:654] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:20.912704 7264 leader_election.cc:343] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.912997 7467 tablet_bootstrap.cc:492] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:20.913421 7467 ts_tablet_manager.cc:1403] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.025s user 0.012s sys 0.001s
I20260317 21:19:20.915784 7467 raft_consensus.cc:359] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.916389 7467 raft_consensus.cc:385] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.916630 7467 raft_consensus.cc:740] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.916963 7182 raft_consensus.cc:3060] T e586f11c37184506a15d726b4eb333be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.913105 7264 leader_election.cc:304] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.917217 7467 consensus_queue.cc:260] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.918159 8592 raft_consensus.cc:2749] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.919664 7467 ts_tablet_manager.cc:1434] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
W20260317 21:19:20.919531 7264 leader_election.cc:343] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.920168 7264 leader_election.cc:304] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:20.920888 8592 raft_consensus.cc:2749] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:20.938728 8614 raft_consensus.cc:493] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.939289 8614 raft_consensus.cc:515] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.941546 8581 raft_consensus.cc:493] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.942505 8614 leader_election.cc:290] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:20.943262 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.942050 8581 raft_consensus.cc:515] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:20.945164 7261 leader_election.cc:343] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.947436 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f0545b61d2d434ea4e51a8f2c4c4d3e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:20.947985 7183 raft_consensus.cc:2468] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:20.949240 6971 leader_election.cc:304] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:20.950153 8579 raft_consensus.cc:2804] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:20.950495 8579 raft_consensus.cc:493] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:20.950798 8579 raft_consensus.cc:3060] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.947059 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569]
I20260317 21:19:20.957300 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_9c887813658144a5ab397c605d8954c0 [id=def26fdbf280448380ad71d8c52b9713]
I20260317 21:19:20.959671 8581 leader_election.cc:290] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:20.978451 8579 raft_consensus.cc:515] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.984063 8591 raft_consensus.cc:493] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:20.984606 8591 raft_consensus.cc:515] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.987673 8591 leader_election.cc:290] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:20.988210 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a44a297079347bab6f8be6abe332bfa" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:20.989869 7261 leader_election.cc:343] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:20.992671 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f0545b61d2d434ea4e51a8f2c4c4d3e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:20.993274 7183 raft_consensus.cc:3060] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:20.993690 7462 tablet_bootstrap.cc:492] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:20.994143 7462 ts_tablet_manager.cc:1403] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.161s user 0.014s sys 0.004s
I20260317 21:19:20.996729 7462 raft_consensus.cc:359] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:20.997416 7462 raft_consensus.cc:385] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:20.997691 7462 raft_consensus.cc:740] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:20.998406 7462 consensus_queue.cc:260] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.005182 7462 ts_tablet_manager.cc:1434] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.011s user 0.007s sys 0.000s
I20260317 21:19:21.008893 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3]
I20260317 21:19:21.010318 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a33d113c213c48728df41846ad097ead [id=df5c9c1007aa4e84aa9363779d23864b]
I20260317 21:19:21.012310 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049]
I20260317 21:19:21.024827 8570 raft_consensus.cc:493] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.025488 8570 raft_consensus.cc:515] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.028455 8570 leader_election.cc:290] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.031385 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73f093d0e8434ca4c41cef6dfe310c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:21.034471 7320 raft_consensus.cc:2468] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:21.009164 8579 leader_election.cc:290] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.060144 7459 ts_tablet_manager.cc:1434] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.277s user 0.006s sys 0.000s
I20260317 21:19:21.075124 7459 tablet_bootstrap.cc:492] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:21.035884 7119 leader_election.cc:304] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.036975 8509 raft_consensus.cc:493] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.079627 8509 raft_consensus.cc:515] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.081180 8570 raft_consensus.cc:2804] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.081851 8570 raft_consensus.cc:493] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.081645 8509 leader_election.cc:290] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.082590 8570 raft_consensus.cc:3060] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.100912 7462 tablet_bootstrap.cc:492] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:21.049921 7183 raft_consensus.cc:2468] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:21.071074 7467 tablet_bootstrap.cc:492] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:21.106349 6971 leader_election.cc:304] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:21.108211 7462 tablet_bootstrap.cc:654] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.111366 7467 tablet_bootstrap.cc:654] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:21.112970 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 77167.2us
W20260317 21:19:21.114195 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 56890.1us
I20260317 21:19:21.109825 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_d6e9cf5141a346f7af807b9f5105eb52 [id=e8f700fb229d465ca87459fd70770227]
I20260317 21:19:21.122217 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:21.123040 7320 raft_consensus.cc:2468] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:21.127954 7467 tablet_bootstrap.cc:492] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:21.128472 7467 ts_tablet_manager.cc:1403] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.058s user 0.008s sys 0.004s
I20260317 21:19:21.131207 7467 raft_consensus.cc:359] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.132120 7467 raft_consensus.cc:385] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.137593 7119 leader_election.cc:304] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
W20260317 21:19:21.144922 6661 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.718s (SENT) [suppressed 1 similar messages]
I20260317 21:19:21.145429 8625 raft_consensus.cc:493] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.145895 8625 raft_consensus.cc:515] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.160388 8625 leader_election.cc:290] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:21.165022 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 49813.3us
I20260317 21:19:21.132375 7467 raft_consensus.cc:740] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:21.172356 7467 consensus_queue.cc:260] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.175228 7467 ts_tablet_manager.cc:1434] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.046s user 0.006s sys 0.001s
I20260317 21:19:21.176393 7467 tablet_bootstrap.cc:492] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:21.149664 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:21.176862 7184 raft_consensus.cc:2468] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:21.179581 8509 raft_consensus.cc:493] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.180207 8509 raft_consensus.cc:515] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.174989 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee]
I20260317 21:19:21.178207 6971 leader_election.cc:304] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:21.182152 6973 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:21.183096 8625 raft_consensus.cc:2804] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.183398 7467 tablet_bootstrap.cc:654] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.183457 8625 raft_consensus.cc:493] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.183804 8625 raft_consensus.cc:3060] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:21.184370 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 46874.2us
I20260317 21:19:21.187058 8615 raft_consensus.cc:2749] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20260317 21:19:21.198120 6661 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 53228.5us
I20260317 21:19:21.200898 8620 raft_consensus.cc:2804] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.201452 8620 raft_consensus.cc:493] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.201849 8620 raft_consensus.cc:3060] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.201020 8509 leader_election.cc:290] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.187865 8579 raft_consensus.cc:2804] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:21.210444 8579 raft_consensus.cc:697] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:21.211303 8579 consensus_queue.cc:237] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.216475 7462 tablet_bootstrap.cc:492] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:21.217096 7462 ts_tablet_manager.cc:1403] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.116s user 0.014s sys 0.002s
I20260317 21:19:21.224795 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf]
I20260317 21:19:21.225828 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:21.226760 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97]
I20260317 21:19:21.227063 7320 raft_consensus.cc:2468] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:21.229125 7119 leader_election.cc:304] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.230567 8509 raft_consensus.cc:2804] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.230948 8509 raft_consensus.cc:493] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.231252 8509 raft_consensus.cc:3060] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.202682 8625 raft_consensus.cc:515] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.236632 8625 leader_election.cc:290] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.206914 7459 tablet_bootstrap.cc:654] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.239198 7462 raft_consensus.cc:359] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.239998 7462 raft_consensus.cc:385] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.240276 7462 raft_consensus.cc:740] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:21.240960 7462 consensus_queue.cc:260] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.242828 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee]
I20260317 21:19:21.247695 6973 leader_election.cc:304] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:21.250003 8619 raft_consensus.cc:2804] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.250572 8619 raft_consensus.cc:493] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.251339 8619 raft_consensus.cc:3060] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.251487 7462 ts_tablet_manager.cc:1434] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.034s user 0.004s sys 0.002s
I20260317 21:19:21.244604 8620 raft_consensus.cc:515] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.257105 8509 raft_consensus.cc:515] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.265646 8509 leader_election.cc:290] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.259361 8570 raft_consensus.cc:515] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.273742 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.274312 7320 raft_consensus.cc:3060] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:21.288503 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 74134.1us
I20260317 21:19:21.291010 7320 raft_consensus.cc:2468] T efd2f45acf114e3ca577103de845c5a1 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:21.285008 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346]
I20260317 21:19:21.293613 7119 leader_election.cc:304] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.295014 8621 raft_consensus.cc:2804] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:21.295437 8621 raft_consensus.cc:697] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:21.296218 8621 consensus_queue.cc:237] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.298732 7467 tablet_bootstrap.cc:492] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:21.299171 7467 ts_tablet_manager.cc:1403] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.123s user 0.006s sys 0.010s
I20260317 21:19:21.302909 8570 leader_election.cc:290] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.303644 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73f093d0e8434ca4c41cef6dfe310c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.304556 8619 raft_consensus.cc:515] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.305437 7320 raft_consensus.cc:3060] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.317653 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_a4ca7b6127064c19b73aee408bc49282 [id=f1ff0d130d854a349d415b1d7d91a836]
I20260317 21:19:21.318920 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_7864164e6f474a35a89094d36d4254ec [id=f2daa1d797d547ff9c87f31e4560581d]
I20260317 21:19:21.326915 8591 raft_consensus.cc:493] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.315229 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:21.320330 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d15d835328d4f7b8fea41cbedf203fe" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:21.327409 8591 raft_consensus.cc:515] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.327943 7183 raft_consensus.cc:3060] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.329638 8591 leader_election.cc:290] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.329942 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73f093d0e8434ca4c41cef6dfe310c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.332573 7032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.334859 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae1b2a549af6485ca3faee3f0ad845a2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:21.336014 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.336846 7033 raft_consensus.cc:2393] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:21.337126 7182 raft_consensus.cc:2393] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate c0ef92fef382401a9564c17e0c223f6a in this term.
W20260317 21:19:21.337675 7261 leader_election.cc:343] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.346158 7183 raft_consensus.cc:2468] T 3d15d835328d4f7b8fea41cbedf203fe P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:21.348704 7467 raft_consensus.cc:359] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.349576 7467 raft_consensus.cc:385] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.349884 7467 raft_consensus.cc:740] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:21.350600 7467 consensus_queue.cc:260] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.352113 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.352711 7329 raft_consensus.cc:3060] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.361878 7467 ts_tablet_manager.cc:1434] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.062s user 0.007s sys 0.000s
I20260317 21:19:21.351433 8620 leader_election.cc:290] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.363989 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:21.364962 8591 raft_consensus.cc:493] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.365433 8591 raft_consensus.cc:515] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.366667 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7d73f093d0e8434ca4c41cef6dfe310c" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:21.367918 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:21.368930 7033 raft_consensus.cc:2393] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:21.367010 7459 tablet_bootstrap.cc:492] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:21.374612 6971 leader_election.cc:304] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:21.375924 8619 leader_election.cc:290] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.376573 8615 raft_consensus.cc:2804] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:21.377287 8615 raft_consensus.cc:697] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:21.378513 8615 consensus_queue.cc:237] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.382284 7462 tablet_bootstrap.cc:492] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:21.375594 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417]
I20260317 21:19:21.386201 8568 raft_consensus.cc:493] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.386689 8568 raft_consensus.cc:515] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.388566 8568 leader_election.cc:290] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.392256 8591 leader_election.cc:290] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.392822 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4af622a9190148ee80b2d5d3d7a92a20" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:21.381534 7459 ts_tablet_manager.cc:1403] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.306s user 0.009s sys 0.008s
W20260317 21:19:21.394104 7261 leader_election.cc:343] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.397027 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3b5825c77ac4d69a39978932051db8c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:21.398762 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f0545b61d2d434ea4e51a8f2c4c4d3e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.399291 7328 raft_consensus.cc:3060] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.401024 7326 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2f0545b61d2d434ea4e51a8f2c4c4d3e" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:21.407725 7321 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:21.409243 7120 leader_election.cc:343] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.409739 7319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e22772281bd74754bb969e87a70d8077" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.411895 8620 consensus_queue.cc:1048] T 722d940aee014df4aeaddbc7aace4f4e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:21.413857 7328 raft_consensus.cc:2468] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:21.417470 7467 tablet_bootstrap.cc:492] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:21.397780 7183 raft_consensus.cc:2468] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:21.423585 7467 tablet_bootstrap.cc:654] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.426236 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58]
I20260317 21:19:21.312572 7320 raft_consensus.cc:2468] T 7d73f093d0e8434ca4c41cef6dfe310c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:21.427516 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c]
I20260317 21:19:21.426460 7329 raft_consensus.cc:2468] T e22772281bd74754bb969e87a70d8077 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:21.427557 7261 leader_election.cc:304] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.428815 7119 leader_election.cc:304] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.429378 8568 raft_consensus.cc:2804] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.429705 8568 raft_consensus.cc:493] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.430014 8568 raft_consensus.cc:3060] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.430405 7459 raft_consensus.cc:359] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.431038 7459 raft_consensus.cc:385] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.431285 7459 raft_consensus.cc:740] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:21.431859 7459 consensus_queue.cc:260] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.449287 8570 raft_consensus.cc:2804] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:21.449755 8570 raft_consensus.cc:697] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:21.450502 8570 consensus_queue.cc:237] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.427937 8591 raft_consensus.cc:493] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.452919 8591 raft_consensus.cc:515] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.454149 7119 leader_election.cc:304] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.456027 8620 raft_consensus.cc:2804] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:21.455998 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f69403b3b811419e87916a381656e1ec" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:21.456589 7120 leader_election.cc:343] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.456622 8620 raft_consensus.cc:697] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
W20260317 21:19:21.457568 7261 leader_election.cc:343] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.457633 8620 consensus_queue.cc:237] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.453326 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_aacec8c3d7ba49f9b6742baf37b423a8 [id=f966bc12b8824b708f7ff3d9f5789eb8]
I20260317 21:19:21.466985 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.467283 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a44a297079347bab6f8be6abe332bfa" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.468465 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae1b2a549af6485ca3faee3f0ad845a2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.479403 7462 tablet_bootstrap.cc:654] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.481652 8592 raft_consensus.cc:493] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.482173 8592 raft_consensus.cc:515] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.482577 8591 leader_election.cc:290] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.483146 8568 raft_consensus.cc:515] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.484251 8592 leader_election.cc:290] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.485597 8568 leader_election.cc:290] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.487370 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3b5825c77ac4d69a39978932051db8c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:21.487990 7182 raft_consensus.cc:3060] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:21.491279 8632 log.cc:927] Time spent T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f: Append to log took a long time: real 0.094s user 0.000s sys 0.000s
I20260317 21:19:21.494765 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4737071c2aec4449a05bead6af7ad32b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:21.497385 7261 leader_election.cc:343] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.506489 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3b5825c77ac4d69a39978932051db8c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.507241 7036 raft_consensus.cc:2468] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:21.508615 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4af622a9190148ee80b2d5d3d7a92a20" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.511660 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f69403b3b811419e87916a381656e1ec" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.521020 7462 tablet_bootstrap.cc:492] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:21.529472 7182 raft_consensus.cc:2468] T f3b5825c77ac4d69a39978932051db8c P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:21.530507 7261 leader_election.cc:304] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:21.531421 8568 raft_consensus.cc:2804] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:21.532346 8568 raft_consensus.cc:697] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:21.533735 8568 consensus_queue.cc:237] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.532948 7459 ts_tablet_manager.cc:1434] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.135s user 0.010s sys 0.003s
I20260317 21:19:21.538234 7459 tablet_bootstrap.cc:492] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:21.538815 7462 ts_tablet_manager.cc:1403] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.157s user 0.013s sys 0.001s
I20260317 21:19:21.536098 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_24ec4c242298468dba3b539b6f272362 [id=fb617cbcbd634696a101417011a3a53a]
I20260317 21:19:21.542891 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4737071c2aec4449a05bead6af7ad32b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.542826 7462 raft_consensus.cc:359] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.543622 7462 raft_consensus.cc:385] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.543924 7462 raft_consensus.cc:740] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:19:21.544466 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 85970.7us
I20260317 21:19:21.544680 7462 consensus_queue.cc:260] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.547585 7462 ts_tablet_manager.cc:1434] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260317 21:19:21.550817 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f3b5825c77ac4d69a39978932051db8c" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:21.551487 7036 raft_consensus.cc:3060] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.558765 7036 raft_consensus.cc:2468] T f3b5825c77ac4d69a39978932051db8c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:21.566550 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3d15d835328d4f7b8fea41cbedf203fe" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.567139 7329 raft_consensus.cc:3060] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.560708 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_147e7f28d4344faf9b6289f6c999b6a9 [id=fd0410563367415d8e73a23c52e8e839]
W20260317 21:19:21.569447 7264 leader_election.cc:343] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.569904 7264 leader_election.cc:304] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.571923 8568 raft_consensus.cc:2749] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:21.578147 7329 raft_consensus.cc:2468] T 3d15d835328d4f7b8fea41cbedf203fe P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:21.590378 7264 leader_election.cc:343] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.590862 7264 leader_election.cc:304] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.591924 8568 raft_consensus.cc:2749] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:21.590893 7120 leader_election.cc:343] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:21.602365 7264 leader_election.cc:343] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.602830 7264 leader_election.cc:304] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:21.603791 7120 leader_election.cc:343] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.603860 8568 raft_consensus.cc:2749] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:21.608575 7264 leader_election.cc:343] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.609018 7264 leader_election.cc:304] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.609975 8568 raft_consensus.cc:2749] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:21.615496 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f]
W20260317 21:19:21.629276 7264 leader_election.cc:343] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.629931 7264 leader_election.cc:304] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.631088 8568 raft_consensus.cc:2749] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:21.633330 7264 leader_election.cc:343] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.633778 7264 leader_election.cc:304] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.635383 8568 raft_consensus.cc:2749] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:21.673688 8292 catalog_manager.cc:679] Loaded metadata for table default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71]
I20260317 21:19:21.674885 7462 tablet_bootstrap.cc:492] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:21.683869 7467 tablet_bootstrap.cc:492] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:21.684331 7467 ts_tablet_manager.cc:1403] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.267s user 0.005s sys 0.011s
I20260317 21:19:21.693440 7462 tablet_bootstrap.cc:654] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.693351 7467 raft_consensus.cc:359] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.695313 7467 raft_consensus.cc:385] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.695633 7467 raft_consensus.cc:740] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:21.696336 7467 consensus_queue.cc:260] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.700853 8619 raft_consensus.cc:493] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.701308 8619 raft_consensus.cc:515] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.702613 7459 tablet_bootstrap.cc:654] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.703155 8619 leader_election.cc:290] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:21.736796 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 71273.9us
I20260317 21:19:21.738543 6973 leader_election.cc:304] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:21.739435 8615 raft_consensus.cc:2749] T e22772281bd74754bb969e87a70d8077 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:21.746150 7467 ts_tablet_manager.cc:1434] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20260317 21:19:21.758160 8568 raft_consensus.cc:493] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.758689 8568 raft_consensus.cc:515] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.761467 8568 leader_election.cc:290] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.765337 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d51e3d4702724484b776a95eb5acf7ef" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:21.766866 7261 leader_election.cc:343] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.768448 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:21.770164 7329 raft_consensus.cc:2468] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:21.783913 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:21.788851 6971 leader_election.cc:343] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.802515 7462 tablet_bootstrap.cc:492] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:21.804764 7462 ts_tablet_manager.cc:1403] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.130s user 0.014s sys 0.000s
I20260317 21:19:21.805066 6973 leader_election.cc:304] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
W20260317 21:19:21.814636 7821 leader_election.cc:336] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779): Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.745s (SENT)
I20260317 21:19:21.815357 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:21.820972 7467 tablet_bootstrap.cc:492] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:21.828050 8646 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260317 21:19:21.810371 7462 raft_consensus.cc:359] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.836344 7462 raft_consensus.cc:385] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.836907 7462 raft_consensus.cc:740] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:21.837661 7462 consensus_queue.cc:260] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.839884 8625 raft_consensus.cc:2804] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:21.850296 8625 raft_consensus.cc:493] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.850660 8625 raft_consensus.cc:3060] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.851392 7459 tablet_bootstrap.cc:492] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:21.832638 8292 tablet_loader.cc:96] loaded metadata for tablet 039d8bed256a43a6b98075b8430cbf3b (table default.Table_e089a694331c4453b9ab3e9643c7b9fb [id=6761934b095549bb86447782a7f7a611])
I20260317 21:19:21.851920 7459 ts_tablet_manager.cc:1403] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.314s user 0.010s sys 0.015s
I20260317 21:19:21.853130 8292 tablet_loader.cc:96] loaded metadata for tablet 047dd2548dbb4c659694b93633b39bf4 (table default.Table_8d558e5446ca41ec87974c9e1cec387e [id=b26d19da9dff470fb59155939f502343])
I20260317 21:19:21.854442 8292 tablet_loader.cc:96] loaded metadata for tablet 054535edcd5041e9b0429c3e7248b1b7 (table default.Table_47f5a50f046e45fb93865004ab8817c7 [id=f0c5f56084b2423494af54928fa7d346])
I20260317 21:19:21.857503 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d51e3d4702724484b776a95eb5acf7ef" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:21.859383 7264 leader_election.cc:343] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.859881 7264 leader_election.cc:304] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.860816 8568 raft_consensus.cc:2749] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:21.867933 7462 ts_tablet_manager.cc:1434] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.063s user 0.007s sys 0.000s
I20260317 21:19:21.869132 7462 tablet_bootstrap.cc:492] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:21.856467 8292 tablet_loader.cc:96] loaded metadata for tablet 06f6a1c205a74045a9f79daf98809b45 (table default.Table_76d879b3858f45bab280f507b3ecec0f [id=8c7668df40c64097bdb16427b811e097])
I20260317 21:19:21.877840 8292 tablet_loader.cc:96] loaded metadata for tablet 0dd4c1efc0624957832c17dc7a4db3f3 (table default.Table_d6e9cf5141a346f7af807b9f5105eb52 [id=e8f700fb229d465ca87459fd70770227])
I20260317 21:19:21.879858 8292 tablet_loader.cc:96] loaded metadata for tablet 0fd3e35e25bf444cbd821ebd9f5ee018 (table default.Table_44edff94b9794d2d806c2219236b3a8d [id=bcbc5e44c4ce46d2b795f6997fa466ce])
I20260317 21:19:21.880895 7462 tablet_bootstrap.cc:654] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:21.881126 8292 tablet_loader.cc:96] loaded metadata for tablet 10942327694142fa82ada834b10befab (table default.Table_2c61fc3bab454eec8eff338bed7a87f5 [id=0303367398284a9897fde69c6375f9e9])
I20260317 21:19:21.882355 8292 tablet_loader.cc:96] loaded metadata for tablet 1179e8f731604aabacaab3090a5f574c (table default.Table_dc3c4bce28a841739bb2887cf3a40e15 [id=84ccb1b467774d92b7f2d3a00bf78994])
I20260317 21:19:21.893247 8640 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.893823 8640 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.895143 8625 raft_consensus.cc:515] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.895893 8640 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.905511 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:21.906164 7183 raft_consensus.cc:2468] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:21.907639 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:21.908272 7329 raft_consensus.cc:2468] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:21.909442 6973 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:21.911587 8642 raft_consensus.cc:2804] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 21:19:21.912159 8642 raft_consensus.cc:493] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:21.912472 8642 raft_consensus.cc:3060] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:21.913749 7459 raft_consensus.cc:359] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.914346 7459 raft_consensus.cc:385] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:21.914582 7459 raft_consensus.cc:740] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:21.918201 6661 leader_election.cc:336] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: RPC error from VoteRequest() call to peer 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779): Timed out: RequestConsensusVote RPC to 127.6.122.125:32779 timed out after 0.718s (SENT)
I20260317 21:19:21.918620 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:21.925238 8292 tablet_loader.cc:96] loaded metadata for tablet 17610bdee98f49c4b1d8b5481867c6e8 (table default.Table_4ba2b776e70f400bb96eedf80ddfc212 [id=0e420ae0b3114bc6ad01a4bb423061b7])
I20260317 21:19:21.915117 7459 consensus_queue.cc:260] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.929562 8292 tablet_loader.cc:96] loaded metadata for tablet 1833d3ee7bc948ffb34418027b72b0ea (table default.Table_a4ca7b6127064c19b73aee408bc49282 [id=f1ff0d130d854a349d415b1d7d91a836])
I20260317 21:19:21.934475 8292 tablet_loader.cc:96] loaded metadata for tablet 18e4b69574e44eb28a735edc64e97542 (table default.Table_532570073e77485ca4f7d839c89cc71f [id=d5b8b1539e0b4bbb8cf6011174f27065])
I20260317 21:19:21.938925 8292 tablet_loader.cc:96] loaded metadata for tablet 1a68e92ca8bf4cd6ae0f4453c06c35a7 (table default.Table_0c987bd2dfe14e3fb5a15492d97199d2 [id=0fcf81d603864206b5d094391f6361d8])
I20260317 21:19:21.941399 8625 leader_election.cc:290] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.943137 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:21.944403 6971 leader_election.cc:343] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.944888 8568 raft_consensus.cc:493] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.945647 8292 tablet_loader.cc:96] loaded metadata for tablet 1ab2868e51e74828bdd3d68ddbb0477e (table default.Table_d3b3ef2c53324b2ba62c7440297657bb [id=cc8571e1fc7a4721a53d0deab165acc0])
I20260317 21:19:21.945344 8568 raft_consensus.cc:515] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.950735 8292 tablet_loader.cc:96] loaded metadata for tablet 203f33157a0d4b82aab44da452aa46dd (table default.Table_e938de6ad6044c138f1a4502dac19c9b [id=406c2e69be6d4b628fbbdaf942c312a6])
I20260317 21:19:21.951743 7459 ts_tablet_manager.cc:1434] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.100s user 0.005s sys 0.004s
I20260317 21:19:21.952984 7459 tablet_bootstrap.cc:492] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:21.945413 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:21.953805 7329 raft_consensus.cc:3060] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:21.953893 8568 leader_election.cc:290] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:21.956849 8647 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260317 21:19:21.957053 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb3a61ed542741d88d89ef6b4c53e9e6" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:21.944877 7467 tablet_bootstrap.cc:654] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:21.960120 7264 leader_election.cc:343] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.961295 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb3a61ed542741d88d89ef6b4c53e9e6" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:21.961514 8292 tablet_loader.cc:96] loaded metadata for tablet 2152e5cbd2ce4159b0901ba2ffe41f40 (table default.Table_8ec60a799e6c4582bf9873c1762dca5c [id=cb01b021f99b45e3ab76affaeb85a857])
I20260317 21:19:21.962658 8292 tablet_loader.cc:96] loaded metadata for tablet 21b1c7a203ef4c22b3a84631cadfa62a (table default.Table_7fee5e45b4aa4a93900162f41e3a3d60 [id=a73a2c0e2c814324ba832290ee403c4a])
W20260317 21:19:21.962841 7261 leader_election.cc:343] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:21.963878 8292 tablet_loader.cc:96] loaded metadata for tablet 21e607674c734eaea207ed454097791d (table default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc])
I20260317 21:19:21.963227 7261 leader_election.cc:304] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:21.965162 8292 tablet_loader.cc:96] loaded metadata for tablet 221c71b908f942ca97f111f857f0335a (table default.Table_5b2a875dd16d4f49b1aaa18981060f82 [id=5cd5dd8f588b448884a78f84bdf75ab9])
I20260317 21:19:21.965379 8568 raft_consensus.cc:2749] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:21.972767 8642 raft_consensus.cc:515] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.981920 7329 raft_consensus.cc:2468] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:21.985466 8620 raft_consensus.cc:493] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.985792 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:21.985949 8620 raft_consensus.cc:515] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.986271 7183 raft_consensus.cc:3060] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:21.987855 8620 leader_election.cc:290] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.985464 8629 raft_consensus.cc:493] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:21.989277 8629 raft_consensus.cc:515] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:21.991111 8629 leader_election.cc:290] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:21.993857 7119 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:21.994395 8642 leader_election.cc:290] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:21.996011 7183 raft_consensus.cc:2468] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 2.
W20260317 21:19:21.996223 7119 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:21.999684 6973 leader_election.cc:304] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:22.002007 7459 tablet_bootstrap.cc:654] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.008385 8615 raft_consensus.cc:2804] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:22.008960 8615 raft_consensus.cc:697] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:22.009727 8615 consensus_queue.cc:237] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.010600 7467 tablet_bootstrap.cc:492] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:22.011010 7467 ts_tablet_manager.cc:1403] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.190s user 0.012s sys 0.002s
I20260317 21:19:22.013496 7467 raft_consensus.cc:359] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.014169 7467 raft_consensus.cc:385] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.014430 7467 raft_consensus.cc:740] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.015308 7467 consensus_queue.cc:260] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.030223 8292 tablet_loader.cc:96] loaded metadata for tablet 2a221dfad830428db2329d6e543f41dc (table default.Table_5ebd4f0e4c284b4caac2e5f0cddf4e07 [id=2d22abe029d348c484e6aff5df8b3d29])
I20260317 21:19:22.032088 8292 tablet_loader.cc:96] loaded metadata for tablet 2ccf1aa8fc9849978c752f1cebccb535 (table default.Table_9c37a7d170904a5f9277a29447a1b58c [id=c2b650f422de4ad08d2759c4f841e690])
I20260317 21:19:22.037065 7467 ts_tablet_manager.cc:1434] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.026s user 0.007s sys 0.000s
W20260317 21:19:22.038164 7120 leader_election.cc:343] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: 323717acdee841aaa1795a3ca851b40f
I20260317 21:19:22.038605 7120 leader_election.cc:304] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:22.039294 8629 raft_consensus.cc:2749] T 323717acdee841aaa1795a3ca851b40f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:22.042198 7120 leader_election.cc:343] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Not found: Tablet not found: a662b365ec4846bf9e9ee243d41cb91a
I20260317 21:19:22.042625 7467 tablet_bootstrap.cc:492] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:22.042694 7120 leader_election.cc:304] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:22.043992 8620 raft_consensus.cc:2749] T a662b365ec4846bf9e9ee243d41cb91a P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:22.051081 6971 leader_election.cc:304] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:22.052240 8615 raft_consensus.cc:2804] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:19:22.052814 8615 raft_consensus.cc:697] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:22.053776 8615 consensus_queue.cc:237] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.062227 7459 tablet_bootstrap.cc:492] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:22.062695 7459 ts_tablet_manager.cc:1403] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.110s user 0.007s sys 0.002s
I20260317 21:19:22.050019 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "88eae1e70bf94ef3ae0aa64706dcb599" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:22.065410 7459 raft_consensus.cc:359] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.066339 7459 raft_consensus.cc:385] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.066875 7459 raft_consensus.cc:740] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.067512 7459 consensus_queue.cc:260] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.070335 7459 ts_tablet_manager.cc:1434] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.007s user 0.008s sys 0.000s
I20260317 21:19:22.063657 7329 raft_consensus.cc:3060] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:22.081473 8292 tablet_loader.cc:96] loaded metadata for tablet 2fd1d8d70e9849fa9e107b2dd3355c58 (table default.Table_a24e99744e6544b696ade1dba045a68a [id=3cfede0a76b448daab974b495b79aa67])
I20260317 21:19:22.082160 7462 tablet_bootstrap.cc:492] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:22.082662 7462 ts_tablet_manager.cc:1403] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.214s user 0.004s sys 0.007s
I20260317 21:19:22.082752 8292 tablet_loader.cc:96] loaded metadata for tablet 3130fb84aeed42c0bba58b970150c983 (table default.Table_3a3ba3d4d60c4441a9f75ff76812f254 [id=ebd361e38f6e4af5839adcea9e7838ee])
I20260317 21:19:22.085309 7462 raft_consensus.cc:359] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.086037 7462 raft_consensus.cc:385] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.086323 7462 raft_consensus.cc:740] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.087163 7462 consensus_queue.cc:260] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.048600 7467 tablet_bootstrap.cc:654] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.084560 8292 tablet_loader.cc:96] loaded metadata for tablet 3180d86564c945218ec4ac5b35634047 (table default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 [id=54eed51395b74f46b69a56cfa73f3724])
I20260317 21:19:22.105578 7459 tablet_bootstrap.cc:492] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:22.108986 7462 ts_tablet_manager.cc:1434] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.026s user 0.006s sys 0.000s
I20260317 21:19:22.118149 7329 raft_consensus.cc:2468] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 2.
I20260317 21:19:22.119562 7459 tablet_bootstrap.cc:654] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.122776 7467 tablet_bootstrap.cc:492] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:22.123242 7467 ts_tablet_manager.cc:1403] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.081s user 0.009s sys 0.000s
I20260317 21:19:22.124291 7462 tablet_bootstrap.cc:492] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:22.126420 7467 raft_consensus.cc:359] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.127130 7467 raft_consensus.cc:385] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.127398 7467 raft_consensus.cc:740] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.128026 7467 consensus_queue.cc:260] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.130540 8292 tablet_loader.cc:96] loaded metadata for tablet 343047b78fb345729c6b66454e2cab40 (table default.Table_7864164e6f474a35a89094d36d4254ec [id=f2daa1d797d547ff9c87f31e4560581d])
I20260317 21:19:22.132530 8292 tablet_loader.cc:96] loaded metadata for tablet 3d15d835328d4f7b8fea41cbedf203fe (table default.Table_71d3b46d9f544087abd87e1fe254c68a [id=f3b3bdb391b240b1815ee2476ff15417])
I20260317 21:19:22.144271 8292 tablet_loader.cc:96] loaded metadata for tablet 3de1582e47f948cfa6c3c0c997ea573f (table default.Table_9967a6acfcf84b2c9759c612f25702e6 [id=790dbba0c4ed469a8d5e208209cb5f39])
I20260317 21:19:22.146986 8292 tablet_loader.cc:96] loaded metadata for tablet 43702e625cb147339bd5871a5ee83e17 (table default.Table_72df81c9b06c4ac9af9af25d9862df96 [id=8ddc0c58c1de4bb2bba0617199fa1982])
I20260317 21:19:22.150432 7467 ts_tablet_manager.cc:1434] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.027s user 0.003s sys 0.004s
I20260317 21:19:22.153666 8625 raft_consensus.cc:493] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.154178 8625 raft_consensus.cc:515] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.156023 8625 leader_election.cc:290] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:22.158195 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fc38947219047b5abdf71af5554a987" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:22.158746 7183 raft_consensus.cc:2468] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:22.159015 7467 tablet_bootstrap.cc:492] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:22.159896 6971 leader_election.cc:304] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
W20260317 21:19:22.162272 7120 consensus_peers.cc:597] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:22.166610 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fc38947219047b5abdf71af5554a987" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:22.167229 7459 tablet_bootstrap.cc:492] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:22.167274 7320 raft_consensus.cc:2468] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:22.167672 7459 ts_tablet_manager.cc:1403] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.062s user 0.005s sys 0.004s
I20260317 21:19:22.172315 8615 raft_consensus.cc:2804] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:22.172734 8615 raft_consensus.cc:493] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:22.173029 8615 raft_consensus.cc:3060] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:22.177243 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 5e080308f93448138a4846ab6bec7dfa (table default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 5e080308f93448138a4846ab6bec7dfa consensus state has no leader
I20260317 21:19:22.179977 7459 raft_consensus.cc:359] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.180704 7459 raft_consensus.cc:385] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.190930 7467 tablet_bootstrap.cc:654] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.193156 8292 tablet_loader.cc:96] loaded metadata for tablet 445b91a6d1294c44aea0935b34be16c0 (table default.Table_66613ac613e64bf6a8f6afd413d96c34 [id=c4f613aff6244864a9346854c1663b07])
I20260317 21:19:22.194478 8292 tablet_loader.cc:96] loaded metadata for tablet 4495961a559549629dedf8ce032abc9b (table default.Table_525a8435bbe14a13a70b022bb2c07f59 [id=539b531cf22a4816b0aff0f3908b0439])
I20260317 21:19:22.200618 8624 consensus_queue.cc:1048] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:22.180958 7459 raft_consensus.cc:740] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.205241 7459 consensus_queue.cc:260] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.207655 7459 ts_tablet_manager.cc:1434] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.040s user 0.006s sys 0.000s
I20260317 21:19:22.207954 7462 tablet_bootstrap.cc:654] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.213675 8292 tablet_loader.cc:96] loaded metadata for tablet 4737071c2aec4449a05bead6af7ad32b (table default.Table_564099aea7ad48349bd3df5beacc7414 [id=90be81f4eaf546dca3dfdf623ab2f552])
I20260317 21:19:22.215658 8292 tablet_loader.cc:96] loaded metadata for tablet 47ce1e486be64aa2803430a33c780348 (table default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e])
I20260317 21:19:22.219913 8615 raft_consensus.cc:515] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.234849 8292 tablet_loader.cc:96] loaded metadata for tablet 4c19bac7772a498f87f48e6095310347 (table default.Table_d86c06b3eb2849ffb26839db4313557e [id=dc2c9168b15c473ebda7e4884631f569])
I20260317 21:19:22.236131 8292 tablet_loader.cc:96] loaded metadata for tablet 4c5fe667d8a44ec6b2c2c235093cf1a6 (table default.Table_a6ed1b24bea44274bcc65ae15bec45f5 [id=d090c2f87f54461f9234fc5f067aad82])
I20260317 21:19:22.237411 8292 tablet_loader.cc:96] loaded metadata for tablet 4f33574134934f048632aae269244368 (table default.Table_b1de6d27253e4eb8b308dc7f2fc0b643 [id=3b5a177f4aa946b68e8bbd4a502701cb])
I20260317 21:19:22.238525 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fc38947219047b5abdf71af5554a987" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:22.239074 7184 raft_consensus.cc:3060] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:22.239184 8292 tablet_loader.cc:96] loaded metadata for tablet 516540c0439d4b75a3ac1689b8964bf7 (table default.Table_6254d435474446bca35e4c78478b186f [id=a796a06b6825482b969b97c9da0ab79d])
I20260317 21:19:22.246191 7184 raft_consensus.cc:2468] T 7fc38947219047b5abdf71af5554a987 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:22.251279 8640 raft_consensus.cc:493] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.252321 8640 raft_consensus.cc:515] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.247972 6971 leader_election.cc:304] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:22.254271 8640 leader_election.cc:290] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:22.254756 8640 raft_consensus.cc:2804] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:22.255422 8640 raft_consensus.cc:697] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:22.256731 8640 consensus_queue.cc:237] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.258481 7467 tablet_bootstrap.cc:492] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:22.258930 7467 ts_tablet_manager.cc:1403] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.100s user 0.006s sys 0.003s
I20260317 21:19:22.261204 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f34026fd2f4575bb86a9dadb733fe0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:22.261480 7467 raft_consensus.cc:359] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.262181 7467 raft_consensus.cc:385] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.262445 7467 raft_consensus.cc:740] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.263087 7467 consensus_queue.cc:260] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.263935 7183 raft_consensus.cc:2468] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:22.266957 8292 tablet_loader.cc:96] loaded metadata for tablet 54e14c247b10477fafc6ccd6a5be3761 (table default.Table_15292a1c98be431b9d3cacbc7e767c75 [id=0f32cb4e70be48c2a548e9e2d4805822])
I20260317 21:19:22.268028 8292 tablet_loader.cc:96] loaded metadata for tablet 5536ae492ac5492b8349cbadd4a156c9 (table default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 [id=b1d87190d48c47288ea093aa65a528a0])
I20260317 21:19:22.267758 8581 consensus_queue.cc:1048] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:22.269173 8581 consensus_queue.cc:1048] T 3d15d835328d4f7b8fea41cbedf203fe P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:22.275930 6971 leader_election.cc:304] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:22.277076 8625 raft_consensus.cc:2804] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:22.277539 8625 raft_consensus.cc:493] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:22.278077 8625 raft_consensus.cc:3060] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:22.283886 7467 ts_tablet_manager.cc:1434] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260317 21:19:22.272827 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f34026fd2f4575bb86a9dadb733fe0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:22.284597 7320 raft_consensus.cc:2468] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:22.295359 7459 tablet_bootstrap.cc:492] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:22.301523 7459 tablet_bootstrap.cc:654] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.309283 8292 tablet_loader.cc:96] loaded metadata for tablet 564a02888d854ea58d7a590cba816b86 (table default.Table_f2b28cd651c443b58e353fb0ccfd33ed [id=22768a64f9394820997448f0c81605ee])
I20260317 21:19:22.309222 8615 leader_election.cc:290] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:22.312011 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7fc38947219047b5abdf71af5554a987" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:22.312562 7328 raft_consensus.cc:3060] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:22.317343 7467 tablet_bootstrap.cc:492] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:22.321813 8292 tablet_loader.cc:96] loaded metadata for tablet 564d41fc13b04ec688fc38a78f55e83d (table default.Table_98c6c8ccb3864ea59aefae46b94eb7a1 [id=4db1fdd255cb400ea274c5dc5e3fbc8d])
I20260317 21:19:22.322075 7328 raft_consensus.cc:2468] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:22.323107 8292 tablet_loader.cc:96] loaded metadata for tablet 57e721ac4b24438b9b2a0e2569027236 (table default.Table_222217e8357d431baa8615150079a999 [id=1e766b4608a743bb9716dbb5d782b3ab])
I20260317 21:19:22.324276 8292 tablet_loader.cc:96] loaded metadata for tablet 585a5f0834614eb1aa4c3e405fa9bbc8 (table default.Table_cab5f9b577f542c7a07d30cfd4c66fdb [id=09c95f07fdb64f619190d34e308ee017])
I20260317 21:19:22.324532 8568 raft_consensus.cc:493] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.324966 8568 raft_consensus.cc:515] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.326884 8568 leader_election.cc:290] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:22.321086 7459 tablet_bootstrap.cc:492] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:22.328995 7459 ts_tablet_manager.cc:1403] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.034s user 0.012s sys 0.001s
I20260317 21:19:22.333036 7467 tablet_bootstrap.cc:654] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.341777 7459 raft_consensus.cc:359] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.342497 7459 raft_consensus.cc:385] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.342779 7459 raft_consensus.cc:740] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.343423 7459 consensus_queue.cc:260] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.348620 7459 ts_tablet_manager.cc:1434] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.019s user 0.003s sys 0.001s
I20260317 21:19:22.350459 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cf1a0922ed54674aaff8b0f49751c13" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:22.351590 7261 leader_election.cc:343] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:22.354207 7462 tablet_bootstrap.cc:492] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:22.354800 7462 ts_tablet_manager.cc:1403] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.231s user 0.011s sys 0.000s
I20260317 21:19:22.357295 7462 raft_consensus.cc:359] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.357998 7462 raft_consensus.cc:385] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.358247 7462 raft_consensus.cc:740] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.328613 8292 tablet_loader.cc:96] loaded metadata for tablet 5be44990b15345e1b827b8e954dd034d (table default.Table_dfe3f4e660b042fca5a8ce85c8816ad4 [id=0642b3d9f69e4a258a46bc790e472d8e])
I20260317 21:19:22.362175 8292 tablet_loader.cc:96] loaded metadata for tablet 5e080308f93448138a4846ab6bec7dfa (table default.Table_7af8e07bbd5b40b8ad18551c8c575d60 [id=1fb5f0d80f6b49c8a7650e9d56aaefe7])
I20260317 21:19:22.367933 7467 tablet_bootstrap.cc:492] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:22.368376 7467 ts_tablet_manager.cc:1403] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.051s user 0.006s sys 0.003s
I20260317 21:19:22.378158 8620 consensus_queue.cc:1048] T efd2f45acf114e3ca577103de845c5a1 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:22.379966 7467 raft_consensus.cc:359] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.380712 7467 raft_consensus.cc:385] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.380965 7467 raft_consensus.cc:740] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.381587 7467 consensus_queue.cc:260] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.383841 7467 ts_tablet_manager.cc:1434] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.015s user 0.005s sys 0.000s
I20260317 21:19:22.384708 8292 tablet_loader.cc:96] loaded metadata for tablet 611ea189ce5d4d3c8c93c18bfd6508e7 (table default.Table_c928476e679c41589639228020653020 [id=25a841c1c6e2466eb130c7fac5dd1e68])
I20260317 21:19:22.391865 7459 tablet_bootstrap.cc:492] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:22.394745 8292 tablet_loader.cc:96] loaded metadata for tablet 613e735b0f414ddf8888be0dfd23ada3 (table default.Table_147e7f28d4344faf9b6289f6c999b6a9 [id=fd0410563367415d8e73a23c52e8e839])
I20260317 21:19:22.397121 8292 tablet_loader.cc:96] loaded metadata for tablet 63a27911e0e545f5811eb075997d0b0f (table default.Table_1011d346d05146d7bb21b4e71f283d3c [id=ff741e0270c34ca99f3d1e3e7c45293f])
I20260317 21:19:22.402911 8568 raft_consensus.cc:493] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.402885 8620 consensus_queue.cc:1048] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:22.403364 8568 raft_consensus.cc:515] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.405387 7467 tablet_bootstrap.cc:492] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:22.405527 8568 leader_election.cc:290] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:22.394940 7462 consensus_queue.cc:260] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.410323 8292 tablet_loader.cc:96] loaded metadata for tablet 69801721e2b7422eba08becf2bcecfa2 (table default.Table_245e4522696f49ee8303db7daf98929f [id=30926b04ead141b79685077be6f457f0])
I20260317 21:19:22.411983 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cf1a0922ed54674aaff8b0f49751c13" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:22.413497 7459 tablet_bootstrap.cc:654] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.416481 7462 ts_tablet_manager.cc:1434] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.061s user 0.005s sys 0.000s
I20260317 21:19:22.419526 8625 raft_consensus.cc:515] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.430630 8615 consensus_queue.cc:1048] T 2f0545b61d2d434ea4e51a8f2c4c4d3e P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:22.432500 8625 leader_election.cc:290] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:22.437685 7462 tablet_bootstrap.cc:492] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:22.454988 7462 tablet_bootstrap.cc:654] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.459619 8292 tablet_loader.cc:96] loaded metadata for tablet 6caf05c79a5f470e8b7796d0e85e36f1 (table default.Table_f9013931d28a41148cb95eda076837f7 [id=f652dc1222a3411db5a54ad8e8927a58])
I20260317 21:19:22.460940 8292 tablet_loader.cc:96] loaded metadata for tablet 6cf1a0922ed54674aaff8b0f49751c13 (table default.Table_edca133386d246bda40142e7ce350141 [id=795579dfe4564220991eb3e7b2760bd0])
I20260317 21:19:22.462025 8292 tablet_loader.cc:96] loaded metadata for tablet 6deece45839849088dad3d873cc925b2 (table default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d])
I20260317 21:19:22.463825 8292 tablet_loader.cc:96] loaded metadata for tablet 6f980bd133da4b04b101e8df99ccca6f (table default.Table_3a13373a17d3475095ae98c63ab87fea [id=c9e66da59d514c8ca23e694e2d818696])
I20260317 21:19:22.469929 7459 tablet_bootstrap.cc:492] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:22.470386 7459 ts_tablet_manager.cc:1403] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.079s user 0.009s sys 0.000s
I20260317 21:19:22.473009 7459 raft_consensus.cc:359] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.477376 7459 raft_consensus.cc:385] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.477684 7459 raft_consensus.cc:740] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.479547 7462 tablet_bootstrap.cc:492] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:22.480038 7462 ts_tablet_manager.cc:1403] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.043s user 0.005s sys 0.009s
I20260317 21:19:22.482892 7462 raft_consensus.cc:359] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.483685 7462 raft_consensus.cc:385] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.484059 7462 raft_consensus.cc:740] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.484992 7462 consensus_queue.cc:260] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.485069 7459 consensus_queue.cc:260] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.486845 7459 ts_tablet_manager.cc:1434] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.016s user 0.003s sys 0.003s
I20260317 21:19:22.488487 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f34026fd2f4575bb86a9dadb733fe0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:22.489153 7183 raft_consensus.cc:3060] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:22.493525 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab5f3dc7836425d917bd09f1e6bfe90" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:22.497611 8292 tablet_loader.cc:96] loaded metadata for tablet 70472ad137f34237a5a67f74c94aa36b (table default.Table_946b81f72ec6451a9aa9d9d9ea8a0e6a [id=70c8c66f0c294cb59a98040c67c1ae93])
I20260317 21:19:22.500607 8292 tablet_loader.cc:96] loaded metadata for tablet 722d940aee014df4aeaddbc7aace4f4e (table default.Table_81205ffa8e804950ae9aff3e9806bb0a [id=903196623798458ba94ad5d9e1af00a3])
I20260317 21:19:22.507714 7462 ts_tablet_manager.cc:1434] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.027s user 0.005s sys 0.001s
I20260317 21:19:22.508499 8292 tablet_loader.cc:96] loaded metadata for tablet 7377c42c199b408daa7f7e730ec5db06 (table default.Table_12f97907f120466093dbd44c0cd929aa [id=ccab17f886b74e94a13a9ad95275089a])
I20260317 21:19:22.508751 7462 tablet_bootstrap.cc:492] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:22.511648 8292 tablet_loader.cc:96] loaded metadata for tablet 74d445b420134ef5aef51f9c8e0f55da (table default.Table_cafc44dd71714a528bda6622dac7997e [id=c11557d0407d4bf08dac964b24545e44])
I20260317 21:19:22.512949 8292 tablet_loader.cc:96] loaded metadata for tablet 76bd0f5369cf400299039e4631ed11b0 (table default.Table_3ba355ed208a4665a3533866f1bc7d38 [id=728955b2cd16428f8c43d4f7cb19bb05])
I20260317 21:19:22.514148 8292 tablet_loader.cc:96] loaded metadata for tablet 788f0ee1f7de47a49c5601c9b1494885 (table default.Table_a33d113c213c48728df41846ad097ead [id=df5c9c1007aa4e84aa9363779d23864b])
I20260317 21:19:22.516029 8292 tablet_loader.cc:96] loaded metadata for tablet 79dc6cb9627d490498950b28fbcbba8f (table default.Table_a86aea973e8f4d2bb2b38b76509e80e1 [id=59c4b25980df473d8e5865e9f582f5a6])
I20260317 21:19:22.517217 8292 tablet_loader.cc:96] loaded metadata for tablet 7b7b54cbff0c4fd5813d197ddbc899cc (table default.Table_d4a5487a2b7e4218b41223f155418e27 [id=b816766f31714663b685cfcc9a9f537a])
I20260317 21:19:22.518230 8292 tablet_loader.cc:96] loaded metadata for tablet 7d19b386f49a420f8561dd9c2f415aa4 (table default.Table_4827867befa741f69f6ab15938cfe436 [id=5b18e88ee5b64c1fb5f7e361c866f44a])
I20260317 21:19:22.528970 7467 tablet_bootstrap.cc:654] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.541980 7462 tablet_bootstrap.cc:654] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:22.545540 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 53188.9us
I20260317 21:19:22.556277 8292 tablet_loader.cc:96] loaded metadata for tablet 84fcdc682c2f4bcdb61ec832ec42ea1b (table default.Table_dc73eaae587c4db2a622ecd0027888ab [id=ecbfb3157d9945d18d67c1820ba58f97])
I20260317 21:19:22.536280 7183 raft_consensus.cc:2468] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:22.563100 7462 tablet_bootstrap.cc:492] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:22.563524 7462 ts_tablet_manager.cc:1403] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.055s user 0.007s sys 0.004s
I20260317 21:19:22.557535 8292 tablet_loader.cc:96] loaded metadata for tablet 866fdd003dd14bb9a837658e9e730f54 (table default.Table_7112fbaf5a99486b95479c3f8f64a8cd [id=9d3c2af086294042b3f2dd8d284a3f96])
I20260317 21:19:22.573474 6971 leader_election.cc:304] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:22.574697 8292 tablet_loader.cc:96] loaded metadata for tablet 86ae75c4ce214954beecfab9275d0ea9 (table default.Table_15fe620085e9499295d76e3a7b3df064 [id=242dcdafc2b74e1ca17839f96e45af88])
I20260317 21:19:22.574775 8624 raft_consensus.cc:2804] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:22.574676 7462 raft_consensus.cc:359] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.575174 8624 raft_consensus.cc:697] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:22.575316 7462 raft_consensus.cc:385] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.575568 7462 raft_consensus.cc:740] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.575994 8624 consensus_queue.cc:237] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.576166 7462 consensus_queue.cc:260] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.593684 8292 tablet_loader.cc:96] loaded metadata for tablet 8a2df0b2a73f4ed4b349efbb4d424afa (table default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a])
I20260317 21:19:22.595047 8292 tablet_loader.cc:96] loaded metadata for tablet 8a44a297079347bab6f8be6abe332bfa (table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b])
I20260317 21:19:22.593472 7462 ts_tablet_manager.cc:1434] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.030s user 0.007s sys 0.001s
I20260317 21:19:22.601547 8292 tablet_loader.cc:96] loaded metadata for tablet 8c214dfdafc9464ea32fffd8223ae990 (table default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193 [id=121b3b215a0b4914a3f18167636ce46e])
I20260317 21:19:22.602593 7462 tablet_bootstrap.cc:492] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:22.602983 8292 tablet_loader.cc:96] loaded metadata for tablet 8d94620882364ed3929ee314814910c9 (table default.Table_38fe3ad6f1bf4796af58d99b53c46560 [id=872cbb89b0d1405e9858d37dc22c05c4])
I20260317 21:19:22.605059 8292 tablet_loader.cc:96] loaded metadata for tablet 932a5eb6cc2848f6a7d7c02ea47e5100 (table default.Table_e3e7da052b2e491e8471865daead13c3 [id=bb9a8a0765794353b53e63ad8b77c5a7])
I20260317 21:19:22.607059 8292 tablet_loader.cc:96] loaded metadata for tablet 9531ac1602894c43b9c165d9a5eddc76 (table default.Table_36d9fab3efe44666a2ce8cde9311325d [id=c5f222a3263a4fc688b7ba7bd132cf32])
I20260317 21:19:22.608093 8292 tablet_loader.cc:96] loaded metadata for tablet 983d205a22b04c8fa5ae45849fc583ab (table default.Table_db13c9846b0049dca4c784a97c434893 [id=305ca4adf2674d4c8cfa2b63e2316871])
W20260317 21:19:22.610137 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 182556us
W20260317 21:19:22.610307 7261 leader_election.cc:343] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:22.613302 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 146443us
I20260317 21:19:22.621737 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f34026fd2f4575bb86a9dadb733fe0" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:22.622294 7329 raft_consensus.cc:3060] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:22.628397 7329 raft_consensus.cc:2468] T 00f34026fd2f4575bb86a9dadb733fe0 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:22.641222 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab5f3dc7836425d917bd09f1e6bfe90" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:22.645120 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 71721.5us
I20260317 21:19:22.653544 8292 tablet_loader.cc:96] loaded metadata for tablet 99c6483618044272890b59f6b7447081 (table default.Table_d6c3cd8934f349da9f5b908ea58fbb31 [id=be58d9833bbf40ada00dce8cb670c74f])
I20260317 21:19:22.655082 8292 tablet_loader.cc:96] loaded metadata for tablet 99dc0fd2382142c88a80fc7eea173efc (table default.Table_42a3f178b5b24f6388658b40093d97aa [id=d19e8fbcd3e44adfa05c861ad27d72fa])
I20260317 21:19:22.656322 8292 tablet_loader.cc:96] loaded metadata for tablet 9a1819f74297406e8ebb28ae847abdd3 (table default.Table_484f7694e0d143c6b75c6403508c9fe8 [id=cec8a98213c6465bbd2f87c9ebfe080b])
I20260317 21:19:22.657500 8292 tablet_loader.cc:96] loaded metadata for tablet 9b6275af1c9c41ae8e4515deabdbdee1 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa])
I20260317 21:19:22.658901 8292 tablet_loader.cc:96] loaded metadata for tablet 9c3b084b04e944d7b919ec1965742e9f (table default.Table_9a942b013ad0470bb7e5ff4149ca0498 [id=b6601cfd883e4746b106817433b9591b])
I20260317 21:19:22.661571 8292 tablet_loader.cc:96] loaded metadata for tablet a37e53e326914575830e30915ad69cc7 (table default.Table_f709509873824f4d86f7018329a3e917 [id=fffe72c025014fc3aa1c83a0ffe6cc71])
I20260317 21:19:22.662006 8666 consensus_queue.cc:1048] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:22.663508 8292 tablet_loader.cc:96] loaded metadata for tablet a4d64c48641b484ab443380808544d2f (table default.Table_aacec8c3d7ba49f9b6742baf37b423a8 [id=f966bc12b8824b708f7ff3d9f5789eb8])
I20260317 21:19:22.664777 8292 tablet_loader.cc:96] loaded metadata for tablet a5207adefc074c229e4020f5c9406c59 (table default.Table_d44ce7d02ab3404389f88fd94db6a9e9 [id=d216b696276b484cbbae383f095c415b])
I20260317 21:19:22.665012 7462 tablet_bootstrap.cc:654] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.686209 8292 tablet_loader.cc:96] loaded metadata for tablet a8a02da14e4c4e0a9ce93e60255c68a9 (table default.Table_b945e0f5a11c4ca0b4a05171094fbfe5 [id=5e990a74096944a09d424c6b9f4e0547])
I20260317 21:19:22.690274 7467 tablet_bootstrap.cc:492] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:22.690697 7467 ts_tablet_manager.cc:1403] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.306s user 0.004s sys 0.008s
I20260317 21:19:22.688128 8292 tablet_loader.cc:96] loaded metadata for tablet aa814dcc2499482eaf0afc5d95bb40f1 (table default.Table_1bd5ef7ffcac4314b52b8df4265dfb67 [id=b4b3b311ab4e427eb75347fc9e68b436])
I20260317 21:19:22.694393 8292 tablet_loader.cc:96] loaded metadata for tablet ad8163635c344d29b8aaa2275539bd22 (table default.Table_1342223726fb480aba4932d06e045aaf [id=06e60ad198054f26b5512c8f53e70753])
I20260317 21:19:22.695631 8292 tablet_loader.cc:96] loaded metadata for tablet ae1797cc7fc346368c11ec68dcac3b2c (table default.Table_32379e31919e4013961f12a0c3692b4b [id=1df22fee10c243b48bbac1dc1ec4cf06])
I20260317 21:19:22.698943 7462 tablet_bootstrap.cc:492] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:22.699396 7462 ts_tablet_manager.cc:1403] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.097s user 0.011s sys 0.000s
I20260317 21:19:22.701896 7462 raft_consensus.cc:359] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.702561 7462 raft_consensus.cc:385] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.702821 7462 raft_consensus.cc:740] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.705487 7462 consensus_queue.cc:260] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.710286 7462 ts_tablet_manager.cc:1434] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.011s user 0.005s sys 0.000s
I20260317 21:19:22.710245 8292 tablet_loader.cc:96] loaded metadata for tablet b03b54b989d04e0990ee3ccfe7a104a4 (table default.Table_0b503a7e5d7a4189b68e5ee5ae0c7f0d [id=10ce77a6ed8a46afbc55ba2c75b873ac])
I20260317 21:19:22.712317 8292 tablet_loader.cc:96] loaded metadata for tablet b1d9d0828d4646b9952b0162a97d61a8 (table default.Table_f95f659a6c96464296d2b1b3bba4af14 [id=b3a37a1953c04314afe940fecdae24af])
I20260317 21:19:22.714329 8292 tablet_loader.cc:96] loaded metadata for tablet b35816e1de7d4b13927848538cdc3e78 (table default.Table_10b2ba08717843c39931baab0bd22525 [id=31ba0dd99ba7446a9b7a91bee04da27a])
I20260317 21:19:22.716293 8292 tablet_loader.cc:96] loaded metadata for tablet b492743dfc4d48398fbae0ee56882aa3 (table default.Table_e65ec8f2488d4ff99eb80d4f4f45f610 [id=b03f3f8e3b9845b4baba7a895e4ff541])
W20260317 21:19:22.719194 7264 leader_election.cc:343] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:22.719944 7264 leader_election.cc:304] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:22.720587 8666 raft_consensus.cc:2749] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:22.724287 8673 consensus_queue.cc:1048] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 21:19:22.729715 8292 tablet_loader.cc:96] loaded metadata for tablet b516c62d56fb43f2a7c73be237cac63b (table default.Table_53241366a970456f98258014850933b1 [id=55857804ed564fa29bf73d6b20c9b072])
I20260317 21:19:22.731081 8292 tablet_loader.cc:96] loaded metadata for tablet b5a797b22cf34483b0061955da6b2634 (table default.Table_6157fa5cda0e4c34817e01c6bdde4904 [id=4a69de09e4be4630994811c7e432c315])
I20260317 21:19:22.732306 8292 tablet_loader.cc:96] loaded metadata for tablet b68293996a25478b81d5202cb61e394a (table default.Table_03cf469e6cf349c4a7dea286f9ae78a8 [id=2dac5c7f7b364d278b68cff649d81b9b])
I20260317 21:19:22.734699 8292 tablet_loader.cc:96] loaded metadata for tablet b76fddff23d04a9a91ea26a366249784 (table default.Table_5661b77c4d044e2ab5f264577eb585ce [id=e2ce44508df044979246581b29fb1049])
I20260317 21:19:22.735911 8292 tablet_loader.cc:96] loaded metadata for tablet b8c9cd4f760f44c88cdd986542755443 (table default.Table_24ec4c242298468dba3b539b6f272362 [id=fb617cbcbd634696a101417011a3a53a])
I20260317 21:19:22.736474 8674 consensus_queue.cc:1048] T e22772281bd74754bb969e87a70d8077 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:22.743065 8674 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.743693 8674 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.746888 8674 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:22.747890 7037 raft_consensus.cc:3060] T 7d73f093d0e8434ca4c41cef6dfe310c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:22.748250 7459 tablet_bootstrap.cc:492] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:22.759908 7459 tablet_bootstrap.cc:654] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.774947 7467 raft_consensus.cc:359] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.775808 7467 raft_consensus.cc:385] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.776116 7467 raft_consensus.cc:740] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.777081 8292 tablet_loader.cc:96] loaded metadata for tablet b9ded59d55424c4990d2936f92893bbe (table default.Table_044bd8a2b853439c8591bbc87743c963 [id=affc94e2f303497b8e16eefdb55c4139])
I20260317 21:19:22.776908 7467 consensus_queue.cc:260] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.782160 8292 tablet_loader.cc:96] loaded metadata for tablet bbc7e853c2474ba9bc6981601fdc820f (table default.Table_b9162607fa9d414e8d894b6d026d45de [id=ebe2e2c2e957468b9ff5675979714dcf])
I20260317 21:19:22.783489 8292 tablet_loader.cc:96] loaded metadata for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2])
I20260317 21:19:22.786226 8674 consensus_queue.cc:1048] T 7d73f093d0e8434ca4c41cef6dfe310c P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:19:22.792996 7264 leader_election.cc:343] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:22.793521 7264 leader_election.cc:304] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:22.794353 8666 raft_consensus.cc:2749] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:22.796667 8666 consensus_queue.cc:1048] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:22.798405 8292 tablet_loader.cc:96] loaded metadata for tablet bdea6939a3e64a279acef3b987e359a3 (table default.Table_39092d36aadb42628dabdb2bf48c1950 [id=831b6f16d49a402e8cc3d53234e050ba])
I20260317 21:19:22.842396 8292 tablet_loader.cc:96] loaded metadata for tablet c936edddda144fd7b0d6467cdc5ac5cc (table default.Table_20cf24578433433a96ad72f0593ca863 [id=405c1ee68fd5448e87ca58a307644859])
W20260317 21:19:22.848879 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 52445.6us
I20260317 21:19:22.854523 8292 tablet_loader.cc:96] loaded metadata for tablet cbef3d2247dd4b3ba7e1a2495403d8ef (table default.Table_61b756f1959b46f393c2a726704e78aa [id=520224705f0d45feb82bb3a96db2cca0])
I20260317 21:19:22.856926 8292 tablet_loader.cc:96] loaded metadata for tablet d05973b63a8c445995ca51d8aec7bcf9 (table default.Table_8730523559834c72ab08f2cec8b6230a [id=df5ae903a4e146299bd9a1fc9f720cf3])
I20260317 21:19:22.857657 7467 ts_tablet_manager.cc:1434] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.167s user 0.003s sys 0.004s
I20260317 21:19:22.858319 8292 tablet_loader.cc:96] loaded metadata for tablet d0769fc5106f441ba7060ed4fa681316 (table default.Table_2cd62a2d2daa4868a58114c426bca181 [id=ab3da0a6fea343d3b2d399b3437fe6f1])
I20260317 21:19:22.858772 7467 tablet_bootstrap.cc:492] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:22.859665 8292 tablet_loader.cc:96] loaded metadata for tablet d2aba7eba6df430e86e003ef3a179912 (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915])
I20260317 21:19:22.866660 7467 tablet_bootstrap.cc:654] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.877512 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:22.879859 7328 raft_consensus.cc:2393] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:22.881434 7467 tablet_bootstrap.cc:492] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:22.881916 7467 ts_tablet_manager.cc:1403] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.005s
I20260317 21:19:22.884227 7467 raft_consensus.cc:359] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.885489 7467 raft_consensus.cc:385] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.885792 7467 raft_consensus.cc:740] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
W20260317 21:19:22.887107 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 69359.7us
I20260317 21:19:22.896224 8675 raft_consensus.cc:493] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.905534 8629 raft_consensus.cc:493] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.906064 8629 raft_consensus.cc:515] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.908200 8629 leader_election.cc:290] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:22.913343 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:22.914127 7033 raft_consensus.cc:2393] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:22.914547 7459 tablet_bootstrap.cc:492] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:22.914968 7459 ts_tablet_manager.cc:1403] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.167s user 0.016s sys 0.014s
I20260317 21:19:22.920938 7462 tablet_bootstrap.cc:492] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:22.921563 7459 raft_consensus.cc:359] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.910730 8675 raft_consensus.cc:515] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.929502 7459 raft_consensus.cc:385] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:22.932425 7462 tablet_bootstrap.cc:654] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:22.933327 7459 raft_consensus.cc:740] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:22.937065 7459 consensus_queue.cc:260] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.950824 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc7e853c2474ba9bc6981601fdc820f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:22.951572 7328 raft_consensus.cc:2468] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:22.951959 8666 raft_consensus.cc:493] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.952817 8666 raft_consensus.cc:515] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.953352 7459 ts_tablet_manager.cc:1434] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.038s user 0.008s sys 0.000s
I20260317 21:19:22.954429 7459 tablet_bootstrap.cc:492] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:22.886466 7467 consensus_queue.cc:260] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.957162 7119 leader_election.cc:304] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:22.956547 8292 tablet_loader.cc:96] loaded metadata for tablet d51e3d4702724484b776a95eb5acf7ef (table default.Table_62af41ad49db4cfc9d735f36dc9c7666 [id=1e9737d709ec448bb692f08f6b5d517f])
I20260317 21:19:22.961017 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:22.961792 7329 raft_consensus.cc:2393] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
I20260317 21:19:22.962026 8666 leader_election.cc:290] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:22.961333 8292 tablet_loader.cc:96] loaded metadata for tablet d69e14125d7d4412869c20759acc2336 (table default.Table_b4a0af0137f043219343ba24745f5077 [id=153f72c3f0b141e5a6d85c8d4d3f2bf9])
I20260317 21:19:22.978236 8292 tablet_loader.cc:96] loaded metadata for tablet d8247f6d37624ca3afcf0702eb6bdd55 (table default.Table_1e952ee93a1d4d7abcd06a732581a9d2 [id=9ec02080c2144394bbe99f211aec30f0])
I20260317 21:19:22.979409 8292 tablet_loader.cc:96] loaded metadata for tablet d85227a2e355424d92e605dd53d9fd14 (table default.Table_9c887813658144a5ab397c605d8954c0 [id=def26fdbf280448380ad71d8c52b9713])
I20260317 21:19:22.990367 8292 tablet_loader.cc:96] loaded metadata for tablet dde40a1c77854adcabc2fff3baaf4c81 (table default.Table_b12dceef6882429eab39af906714c5e1 [id=3c2963586340401196c86489da629479])
I20260317 21:19:22.990461 8666 raft_consensus.cc:493] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:22.990996 8666 raft_consensus.cc:515] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:22.993530 8666 leader_election.cc:290] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:22.996709 7459 tablet_bootstrap.cc:654] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.004597 8292 tablet_loader.cc:96] loaded metadata for tablet dfff4c8478b64436a666a14c133c36d3 (table default.Table_1ce0835b11b44dcc9535d7cf85dead34 [id=a5b88a1e24b34d11906770f114c31ab0])
I20260317 21:19:23.006364 8292 tablet_loader.cc:96] loaded metadata for tablet e06a1488c60f4d1387417ce30579fbf4 (table default.Table_54698fcc6a7a46deaba83055d4a1c78d [id=3e906ea0e8af47e383d0403c155be644])
I20260317 21:19:23.008214 8292 tablet_loader.cc:96] loaded metadata for tablet e1094068903840a8ab5d07c7fb0635bf (table default.Table_08bd63c86aca4345bc3ca2746d45456e [id=93623d5786f645c7a1ad496e656caa24])
I20260317 21:19:23.010294 8666 raft_consensus.cc:493] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.010778 8666 raft_consensus.cc:515] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.012037 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69801721e2b7422eba08becf2bcecfa2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.014333 8666 leader_election.cc:290] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:22.936575 8675 leader_election.cc:290] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.020881 7183 raft_consensus.cc:1275] T 88eae1e70bf94ef3ae0aa64706dcb599 P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Refusing update from remote peer d3cb3b49e3e94537a6e434ea4e037c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 21:19:23.021703 8629 raft_consensus.cc:493] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:19:23.021917 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 64854us
I20260317 21:19:23.022197 8691 raft_consensus.cc:493] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.022172 8629 raft_consensus.cc:515] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.022894 7467 ts_tablet_manager.cc:1434] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.141s user 0.004s sys 0.001s
I20260317 21:19:23.022661 8691 raft_consensus.cc:515] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.023382 8620 raft_consensus.cc:2804] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:23.023756 8620 raft_consensus.cc:493] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:23.023890 7467 tablet_bootstrap.cc:492] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:23.024081 8620 raft_consensus.cc:3060] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.024140 8629 leader_election.cc:290] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.026389 8624 consensus_queue.cc:1048] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:23.027004 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69801721e2b7422eba08becf2bcecfa2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:23.027222 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5207adefc074c229e4020f5c9406c59" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:23.027725 7329 raft_consensus.cc:2468] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:23.028887 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:23.030854 7261 leader_election.cc:343] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.033681 7119 leader_election.cc:304] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:23.035034 7459 tablet_bootstrap.cc:492] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:23.035483 7459 ts_tablet_manager.cc:1403] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.081s user 0.010s sys 0.000s
I20260317 21:19:23.041980 8691 leader_election.cc:290] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.042289 7032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5027b892229491eb5a829cacd29a700" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.042250 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4495961a559549629dedf8ce032abc9b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:23.043855 7261 leader_election.cc:343] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.043772 7032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4495961a559549629dedf8ce032abc9b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.036103 8673 raft_consensus.cc:2804] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:23.045666 8673 raft_consensus.cc:493] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:23.046375 8673 raft_consensus.cc:3060] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.051491 7329 raft_consensus.cc:1275] T 88eae1e70bf94ef3ae0aa64706dcb599 P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Refusing update from remote peer d3cb3b49e3e94537a6e434ea4e037c3f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 21:19:23.052688 8292 tablet_loader.cc:96] loaded metadata for tablet e22772281bd74754bb969e87a70d8077 (table default.Table_ac0ee103335146a3b4168471de3fd782 [id=dc1cb0c144ba49c495561aa2494bf4d2])
I20260317 21:19:23.055466 7120 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:23.056427 8292 tablet_loader.cc:96] loaded metadata for tablet e2c52f4843234e8584a66e5e8e065667 (table default.Table_3bfb11e1f4714c45b0b46b83c6eac620 [id=51c4378b9555433284ae022a04e064d2])
I20260317 21:19:23.057742 8292 tablet_loader.cc:96] loaded metadata for tablet e4522181cc434203ac14c9b07587fec8 (table default.Table_d4e37f648814488a8969d384ba941f06 [id=577b3aea931d4e518d588b81917b49c8])
I20260317 21:19:23.063575 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5027b892229491eb5a829cacd29a700" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:23.065259 7462 tablet_bootstrap.cc:492] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
W20260317 21:19:23.065354 7261 leader_election.cc:343] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.065735 7462 ts_tablet_manager.cc:1403] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.145s user 0.006s sys 0.007s
I20260317 21:19:23.068987 7462 raft_consensus.cc:359] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.069655 7462 raft_consensus.cc:385] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.070367 7462 raft_consensus.cc:740] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.054484 7459 raft_consensus.cc:359] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.076949 7459 raft_consensus.cc:385] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.077277 7459 raft_consensus.cc:740] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.071120 7462 consensus_queue.cc:260] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.090832 7462 ts_tablet_manager.cc:1434] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.025s user 0.006s sys 0.000s
I20260317 21:19:23.090483 7459 consensus_queue.cc:260] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:23.103711 7264 leader_election.cc:343] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.104199 7264 leader_election.cc:304] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:23.105708 8702 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:23.106398 8702 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:23.106909 8666 raft_consensus.cc:2749] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.059150 8292 tablet_loader.cc:96] loaded metadata for tablet e586f11c37184506a15d726b4eb333be (table default.Table_b6fea0d08b2b48609aed18c1983a38a7 [id=882a641cd0144f698f638abbdf0f76a3])
I20260317 21:19:23.119869 7462 tablet_bootstrap.cc:492] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:23.120834 8292 tablet_loader.cc:96] loaded metadata for tablet e988430ec1074531b0e173e96983a0cb (table default.Table_8f61c0b1dfa846baa3c30adcbba8b683 [id=4a759617375643bcbf00a8b9ddcf0677])
I20260317 21:19:23.122997 8292 tablet_loader.cc:96] loaded metadata for tablet eb3a61ed542741d88d89ef6b4c53e9e6 (table default.Table_e48e213b99454430b7ede635351ef076 [id=13df2f7b70f6435db6c5cb826b003373])
I20260317 21:19:23.072269 8629 raft_consensus.cc:3060] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.127766 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "efd2f45acf114e3ca577103de845c5a1" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:23.127156 7264 leader_election.cc:343] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.129532 7264 leader_election.cc:304] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:23.130259 8666 raft_consensus.cc:2749] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.139161 7459 ts_tablet_manager.cc:1434] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.103s user 0.009s sys 0.001s
I20260317 21:19:23.140945 8702 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:23.143469 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
W20260317 21:19:23.156032 7264 leader_election.cc:343] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.156675 7264 leader_election.cc:304] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:23.157770 8666 raft_consensus.cc:2749] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.163076 7459 tablet_bootstrap.cc:492] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:23.124315 8292 tablet_loader.cc:96] loaded metadata for tablet ec5aa338aa2a41f0a7aeb4194189208f (table default.Table_760f5ff0097f4ae6aa93439b05ef00c4 [id=ed796ac0de91479f867865bd0a15f4ee])
I20260317 21:19:23.169310 8292 tablet_loader.cc:96] loaded metadata for tablet edd09d2a9ba741bc94ac682018337a83 (table default.Table_d821b351490e4553b71a09c4055e26cd [id=87c45aeb0b21447bac6301a6b5845376])
I20260317 21:19:23.171324 8292 tablet_loader.cc:96] loaded metadata for tablet efd2f45acf114e3ca577103de845c5a1 (table default.Table_29eda40c73594995aebde51a209340f9 [id=f70919d6683e4c3f8f6a9a420ca49d9c])
I20260317 21:19:23.173074 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc7e853c2474ba9bc6981601fdc820f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.173919 7035 raft_consensus.cc:2468] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:23.175825 8620 raft_consensus.cc:515] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.177623 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8744f6237e154dd99bc4c7e54822f7be" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.180727 8673 raft_consensus.cc:515] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.182912 8673 leader_election.cc:290] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.188354 8629 raft_consensus.cc:2749] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.189741 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5207adefc074c229e4020f5c9406c59" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.192764 8620 leader_election.cc:290] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.193588 7459 tablet_bootstrap.cc:654] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.198891 8292 tablet_loader.cc:96] loaded metadata for tablet f044d4c56cf24215a7f0beec13ece53a (table default.Table_1cc7099df8b34e2ba85b14761993b230 [id=1903b54d3b9a49ffb0d5f5e1f7bb9c05])
I20260317 21:19:23.214108 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc7e853c2474ba9bc6981601fdc820f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:23.214696 7329 raft_consensus.cc:3060] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.205631 7467 tablet_bootstrap.cc:654] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.222536 7329 raft_consensus.cc:2468] T bbc7e853c2474ba9bc6981601fdc820f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:23.223582 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5207adefc074c229e4020f5c9406c59" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:23.224328 7320 raft_consensus.cc:3060] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.233224 7462 tablet_bootstrap.cc:654] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.236663 7119 leader_election.cc:304] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:23.237210 8292 tablet_loader.cc:96] loaded metadata for tablet f5027b892229491eb5a829cacd29a700 (table default.Table_fa44c16b78054e6fb846b384a1c68805 [id=696cfb971ec54750aa0c70ddaa40610e])
I20260317 21:19:23.238504 8292 tablet_loader.cc:96] loaded metadata for tablet f69403b3b811419e87916a381656e1ec (table default.Table_579fde97e6404c37adf7e33736201ea4 [id=c59cee028a4b47629c748c9242bf28a2])
I20260317 21:19:23.238546 8620 raft_consensus.cc:2804] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:23.238982 8620 raft_consensus.cc:697] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:23.239636 8292 tablet_loader.cc:96] loaded metadata for tablet f6cd3f64f9b94d089b8ecf2462007ee6 (table default.Table_c037d6417fdf415cb20c3bb9c342c07d [id=10eb9e3956be42aab8e5bfc43d9b9e4b])
I20260317 21:19:23.254341 7459 tablet_bootstrap.cc:492] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:23.239770 8620 consensus_queue.cc:237] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.254809 7459 ts_tablet_manager.cc:1403] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.092s user 0.010s sys 0.001s
I20260317 21:19:23.263077 7459 raft_consensus.cc:359] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.263764 7459 raft_consensus.cc:385] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.264021 7459 raft_consensus.cc:740] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.265990 7120 leader_election.cc:304] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:23.267163 8674 raft_consensus.cc:3060] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.270016 8292 tablet_loader.cc:96] loaded metadata for tablet f7ec20308a9244a49992e378fa7b4f6c (table default.Table_bb6c327aff234752a4f6930f24dc07b9 [id=62b1423a48c64043bd48fb864601b793])
I20260317 21:19:23.271595 8292 tablet_loader.cc:96] loaded metadata for tablet f9e98c10a83e429a98a0fb4328e1b0ee (table default.Table_4b718b3df61e49f397680a56009f72ea [id=802599f023ff413d9c34acb459b85f48])
I20260317 21:19:23.275766 8674 raft_consensus.cc:2749] T 8744f6237e154dd99bc4c7e54822f7be P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.266002 8692 consensus_queue.cc:1048] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.011s
I20260317 21:19:23.264617 7459 consensus_queue.cc:260] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.285305 7459 ts_tablet_manager.cc:1434] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.030s user 0.003s sys 0.005s
I20260317 21:19:23.285874 7320 raft_consensus.cc:2468] T a5207adefc074c229e4020f5c9406c59 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:23.286455 7459 tablet_bootstrap.cc:492] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:23.291990 8625 consensus_queue.cc:1048] T 88eae1e70bf94ef3ae0aa64706dcb599 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:23.294749 8292 tablet_loader.cc:96] loaded metadata for tablet fd7cf19121b5496a849118de4a29db5b (table default.Table_dad8a2efd2824c4e81314759e5700d97 [id=6de336aaef3744f58b4fdfab525c8f2f])
I20260317 21:19:23.296129 8292 tablet_loader.cc:96] loaded metadata for tablet fe51fd1fdd864b6f8b44cb45ba332c4d (table default.Table_846f03c95b0340468fae274b89fcf350 [id=1e920e97afb04ddb90f4ab7223496d19])
I20260317 21:19:23.296165 8691 consensus_queue.cc:1048] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:23.306165 8691 consensus_queue.cc:1048] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:19:23.309796 6660 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet b8c9cd4f760f44c88cdd986542755443 (table default.Table_24ec4c242298468dba3b539b6f272362 [id=fb617cbcbd634696a101417011a3a53a], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet b8c9cd4f760f44c88cdd986542755443 consensus state has no leader
I20260317 21:19:23.312429 8691 consensus_queue.cc:1048] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:23.323767 8694 raft_consensus.cc:493] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.324271 8694 raft_consensus.cc:515] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.325480 8292 tablet_loader.cc:96] loaded metadata for tablet ff04afc5dc834ac4a51f97e29e4cc96e (table default.Table_eb6dce2e8eef477681d82e18ab326537 [id=8b0cf1fd0865413cb5ce15157e7eb845])
I20260317 21:19:23.326332 8694 leader_election.cc:290] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.324594 6973 leader_election.cc:304] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:23.329375 8691 raft_consensus.cc:2749] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.341770 7459 tablet_bootstrap.cc:654] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.342108 7462 tablet_bootstrap.cc:492] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:23.342559 7462 ts_tablet_manager.cc:1403] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.223s user 0.005s sys 0.006s
I20260317 21:19:23.342711 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70560dde990843e0b7c74ae3206c58e4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:23.344487 7120 leader_election.cc:343] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:23.345947 6971 leader_election.cc:343] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.344959 7119 leader_election.cc:304] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:23.351198 8674 raft_consensus.cc:2804] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:23.351706 8674 raft_consensus.cc:697] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:23.352591 8674 consensus_queue.cc:237] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.366811 7462 raft_consensus.cc:359] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.371170 7459 tablet_bootstrap.cc:492] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:23.371635 7459 ts_tablet_manager.cc:1403] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.085s user 0.007s sys 0.002s
I20260317 21:19:23.371592 7462 raft_consensus.cc:385] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.371994 7462 raft_consensus.cc:740] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.372679 7462 consensus_queue.cc:260] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.374328 7459 raft_consensus.cc:359] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.375053 7459 raft_consensus.cc:385] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.375336 7459 raft_consensus.cc:740] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.376068 7459 consensus_queue.cc:260] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.399581 7459 ts_tablet_manager.cc:1434] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20260317 21:19:23.406891 8704 raft_consensus.cc:493] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.407485 8704 raft_consensus.cc:515] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.410293 8704 leader_election.cc:290] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.412341 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f427b15a7b2a47e18271e8884c637f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:23.413980 7329 raft_consensus.cc:2468] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:23.418298 7459 tablet_bootstrap.cc:492] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:23.422194 7467 tablet_bootstrap.cc:492] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:23.422734 7467 ts_tablet_manager.cc:1403] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.399s user 0.015s sys 0.011s
I20260317 21:19:23.419283 7119 leader_election.cc:304] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:23.427561 7462 ts_tablet_manager.cc:1434] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.085s user 0.008s sys 0.000s
I20260317 21:19:23.429131 7462 tablet_bootstrap.cc:492] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:23.431085 8620 raft_consensus.cc:2804] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:23.431726 8620 raft_consensus.cc:493] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:23.432370 8620 raft_consensus.cc:3060] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.451578 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70560dde990843e0b7c74ae3206c58e4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:23.452279 7322 raft_consensus.cc:2468] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
W20260317 21:19:23.482523 8292 catalog_manager.cc:1486] Time spent T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Loading table and tablet metadata into memory: real 3.842s user 0.598s sys 0.070s
I20260317 21:19:23.482832 8292 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:19:23.486397 6749 rpcz_store.cc:275] Call kudu.master.MasterService.ConnectToMaster from 127.0.0.1:50358 (request call id 35) took 3802 ms. Trace:
I20260317 21:19:23.487100 6749 rpcz_store.cc:276] 0317 21:19:19.683842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:19.683982 (+ 140us) service_pool.cc:225] Handling call
0317 21:19:23.486292 (+3802310us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:23.488219 6753 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:23.489128 6753 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 14) took 2419 ms (client timeout 745 ms). Trace:
W20260317 21:19:23.489480 6753 rpcz_store.cc:269] 0317 21:19:21.069530 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:21.069671 (+ 141us) service_pool.cc:225] Handling call
0317 21:19:23.489053 (+2419382us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:23.490352 6755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:23.491207 6755 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32962 (request call id 387) took 3067 ms (client timeout 718 ms). Trace:
W20260317 21:19:23.491598 6755 rpcz_store.cc:269] 0317 21:19:20.424073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:20.424230 (+ 157us) service_pool.cc:225] Handling call
0317 21:19:23.491136 (+3066906us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:23.484454 6758 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:23.495776 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:23.496835 8702 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260317 21:19:23.425199 7467 raft_consensus.cc:359] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.496351 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:23.495033 6756 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 423 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
W20260317 21:19:23.501286 6756 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:32962 (request call id 388) took 2302 ms (client timeout 718 ms). Trace:
W20260317 21:19:23.501637 6756 rpcz_store.cc:269] 0317 21:19:21.199347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:21.199488 (+ 141us) service_pool.cc:225] Handling call
0317 21:19:23.501212 (+2301724us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20260317 21:19:23.502312 6751 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:46752 (request call id 13) took 3661 ms (client timeout 745 ms). Trace:
I20260317 21:19:23.502713 7462 tablet_bootstrap.cc:654] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:23.502638 6751 rpcz_store.cc:269] 0317 21:19:19.841393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:19.841526 (+ 133us) service_pool.cc:225] Handling call
0317 21:19:23.499308 (+3657782us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20260317 21:19:23.505616 8620 raft_consensus.cc:515] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.510860 7467 raft_consensus.cc:385] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.511222 7467 raft_consensus.cc:740] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.511957 7467 consensus_queue.cc:260] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.513326 7032 raft_consensus.cc:3060] T efd2f45acf114e3ca577103de845c5a1 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.513670 7459 tablet_bootstrap.cc:654] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.520562 8620 leader_election.cc:290] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.524694 8718 raft_consensus.cc:493] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.525458 8718 raft_consensus.cc:515] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.527055 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f427b15a7b2a47e18271e8884c637f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:23.528172 7322 raft_consensus.cc:3060] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.536504 8718 leader_election.cc:290] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:23.538365 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829bc3959a9e4c2dbb6277cde5969b0e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:23.540481 7261 leader_election.cc:343] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.547946 7462 tablet_bootstrap.cc:492] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:23.548485 7462 ts_tablet_manager.cc:1403] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.120s user 0.005s sys 0.006s
I20260317 21:19:23.550840 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a5207adefc074c229e4020f5c9406c59" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:23.551394 7033 raft_consensus.cc:3060] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.552003 7462 raft_consensus.cc:359] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.559226 7033 raft_consensus.cc:2468] T a5207adefc074c229e4020f5c9406c59 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:23.552700 7462 raft_consensus.cc:385] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.560779 7462 raft_consensus.cc:740] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:19:23.583402 8720 log.cc:927] Time spent T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Append to log took a long time: real 0.084s user 0.001s sys 0.000s
I20260317 21:19:23.589352 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f427b15a7b2a47e18271e8884c637f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:23.590013 7037 raft_consensus.cc:2468] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:23.592746 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f427b15a7b2a47e18271e8884c637f4e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:23.593365 7037 raft_consensus.cc:3060] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.598400 7459 tablet_bootstrap.cc:492] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:23.598881 7459 ts_tablet_manager.cc:1403] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.199s user 0.006s sys 0.005s
I20260317 21:19:23.607996 7467 ts_tablet_manager.cc:1434] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.185s user 0.005s sys 0.000s
I20260317 21:19:23.561543 7462 consensus_queue.cc:260] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.609248 7467 tablet_bootstrap.cc:492] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:23.617655 7033 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbc7e853c2474ba9bc6981601fdc820f" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:23.618389 7033 raft_consensus.cc:3060] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.628881 7037 raft_consensus.cc:2468] T f427b15a7b2a47e18271e8884c637f4e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:23.650904 7322 raft_consensus.cc:2468] T f427b15a7b2a47e18271e8884c637f4e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:23.656585 7119 leader_election.cc:304] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:23.663810 7459 raft_consensus.cc:359] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.664820 7459 raft_consensus.cc:385] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.665136 7459 raft_consensus.cc:740] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.674530 7462 ts_tablet_manager.cc:1434] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.126s user 0.003s sys 0.004s
I20260317 21:19:23.675735 7462 tablet_bootstrap.cc:492] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:23.683177 8625 consensus_queue.cc:1048] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:23.685834 8704 raft_consensus.cc:2804] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:23.693290 8704 raft_consensus.cc:697] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:23.694851 8704 consensus_queue.cc:237] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.704113 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "829bc3959a9e4c2dbb6277cde5969b0e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
W20260317 21:19:23.704356 7119 consensus_peers.cc:597] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3d40549347f8463797d6791136883371. This is attempt 6: this message will repeat every 5th retry.
I20260317 21:19:23.665812 7459 consensus_queue.cc:260] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.710906 7459 ts_tablet_manager.cc:1434] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.112s user 0.005s sys 0.002s
I20260317 21:19:23.711943 7459 tablet_bootstrap.cc:492] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:23.717183 7462 tablet_bootstrap.cc:654] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.730942 7033 raft_consensus.cc:2468] T bbc7e853c2474ba9bc6981601fdc820f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:23.733532 6973 leader_election.cc:304] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:23.749828 8625 raft_consensus.cc:2804] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:23.751052 8625 raft_consensus.cc:493] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:23.751506 8625 raft_consensus.cc:3060] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.753340 7459 tablet_bootstrap.cc:654] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.775552 8292 catalog_manager.cc:1269] Loaded cluster ID: e5fe7ace045a4a159f8190ca296ef5bd
I20260317 21:19:23.775879 8292 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:19:23.791083 7462 tablet_bootstrap.cc:492] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:23.791507 7462 ts_tablet_manager.cc:1403] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.116s user 0.006s sys 0.005s
I20260317 21:19:23.793967 7462 raft_consensus.cc:359] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.794613 7462 raft_consensus.cc:385] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.794905 7462 raft_consensus.cc:740] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.795531 7462 consensus_queue.cc:260] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:23.828353 7264 leader_election.cc:343] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:23.828857 7264 leader_election.cc:304] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:23.830252 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 126233us
I20260317 21:19:23.833388 8666 raft_consensus.cc:2749] T 829bc3959a9e4c2dbb6277cde5969b0e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:23.845110 7462 ts_tablet_manager.cc:1434] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.053s user 0.004s sys 0.002s
I20260317 21:19:23.850726 7459 tablet_bootstrap.cc:492] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:23.851169 7459 ts_tablet_manager.cc:1403] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.139s user 0.010s sys 0.000s
I20260317 21:19:23.819097 8625 raft_consensus.cc:515] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.858968 8625 leader_election.cc:290] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.865419 7459 raft_consensus.cc:359] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.866248 7459 raft_consensus.cc:385] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:23.866526 7459 raft_consensus.cc:740] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:23.867197 7459 consensus_queue.cc:260] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.869082 7467 tablet_bootstrap.cc:654] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.873778 8666 raft_consensus.cc:493] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.874387 8666 raft_consensus.cc:515] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.878803 8666 leader_election.cc:290] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:23.883646 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b7b54cbff0c4fd5813d197ddbc899cc" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:23.887233 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70560dde990843e0b7c74ae3206c58e4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:23.887990 7179 raft_consensus.cc:3060] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.902665 8619 raft_consensus.cc:493] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:23.903266 8619 raft_consensus.cc:515] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:23.904924 7459 ts_tablet_manager.cc:1434] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.053s user 0.005s sys 0.003s
I20260317 21:19:23.905929 8619 leader_election.cc:290] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:23.913055 7462 tablet_bootstrap.cc:492] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:23.920768 8292 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:19:23.929414 7459 tablet_bootstrap.cc:492] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:23.919790 7462 tablet_bootstrap.cc:654] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:23.924307 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70560dde990843e0b7c74ae3206c58e4" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:23.937525 7320 raft_consensus.cc:3060] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:23.930286 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:23.941401 7328 raft_consensus.cc:2468] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:23.954149 7320 raft_consensus.cc:2468] T 70560dde990843e0b7c74ae3206c58e4 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:23.957453 8292 catalog_manager.cc:6055] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Loaded TSK: 0
I20260317 21:19:23.958441 8292 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:19:23.964505 7467 tablet_bootstrap.cc:492] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:23.965085 7467 ts_tablet_manager.cc:1403] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.356s user 0.012s sys 0.000s
I20260317 21:19:23.989526 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7b7b54cbff0c4fd5813d197ddbc899cc" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.000545 8619 consensus_queue.cc:1048] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:24.014581 7462 tablet_bootstrap.cc:492] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:24.015020 7462 ts_tablet_manager.cc:1403] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.102s user 0.009s sys 0.000s
I20260317 21:19:24.017588 7462 raft_consensus.cc:359] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.018218 7462 raft_consensus.cc:385] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.018483 7462 raft_consensus.cc:740] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.021199 7459 tablet_bootstrap.cc:654] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:24.022277 6973 leader_election.cc:304] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:24.023065 8624 raft_consensus.cc:2804] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:24.023383 8624 raft_consensus.cc:493] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:24.023706 8624 raft_consensus.cc:3060] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.019062 7462 consensus_queue.cc:260] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.029295 7462 ts_tablet_manager.cc:1434] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.014s user 0.007s sys 0.000s
W20260317 21:19:24.037036 7261 leader_election.cc:343] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.038461 8666 raft_consensus.cc:493] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.038900 8666 raft_consensus.cc:515] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.041596 8666 leader_election.cc:290] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.050695 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.054921 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:24.064499 6971 leader_election.cc:343] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.048671 7179 raft_consensus.cc:2468] T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
W20260317 21:19:24.075114 7261 leader_election.cc:343] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.078788 6749 master_service.cc:954] ReplaceTablet: received request to replace tablet 8a44a297079347bab6f8be6abe332bfa from {username='slave'} at 127.0.0.1:50358
I20260317 21:19:24.084898 7467 raft_consensus.cc:359] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.088323 7467 raft_consensus.cc:385] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.088897 7467 raft_consensus.cc:740] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.090492 6971 leader_election.cc:304] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:24.092373 8692 raft_consensus.cc:2804] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:24.094599 8692 raft_consensus.cc:697] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:24.095767 8692 consensus_queue.cc:237] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.102491 8666 raft_consensus.cc:493] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.103070 8666 raft_consensus.cc:515] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.105623 8666 leader_election.cc:290] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:24.107061 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:24.108811 7261 leader_election.cc:343] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.120208 6743 master_service.cc:954] ReplaceTablet: received request to replace tablet 8a44a297079347bab6f8be6abe332bfa from {username='slave'} at 127.0.0.1:50358
I20260317 21:19:24.122695 6750 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50358:
table { table_name: "default.Table_e65ec8f2488d4ff99eb80d4f4f45f610" } modify_external_catalogs: true
I20260317 21:19:24.123449 6750 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50358:
table { table_name: "default.Table_e65ec8f2488d4ff99eb80d4f4f45f610" } modify_external_catalogs: true
W20260317 21:19:24.129066 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 108853us
I20260317 21:19:24.129508 6744 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:50358:
table { table_name: "default.Table_d86c06b3eb2849ffb26839db4313557e" } modify_external_catalogs: true
I20260317 21:19:24.130084 6744 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:50358:
table { table_name: "default.Table_d86c06b3eb2849ffb26839db4313557e" } modify_external_catalogs: true
I20260317 21:19:24.134274 6745 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50358:
name: "default.Table_3b1322b6b52a474ba8475816df4c8ee7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:24.138936 6745 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3b1322b6b52a474ba8475816df4c8ee7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:24.141379 6747 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50358:
name: "default.Table_acfbab364a9442afbd75bcbdfb2b785c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:19:24.143193 6747 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_acfbab364a9442afbd75bcbdfb2b785c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:24.145462 6746 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:50358:
name: "default.Table_7d9bd016aa3340c38dc1ecba85639bee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:19:24.146373 8624 raft_consensus.cc:515] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:24.147569 6746 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d9bd016aa3340c38dc1ecba85639bee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:19:24.149477 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:24.151224 6971 leader_election.cc:343] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.101512 7467 consensus_queue.cc:260] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.156270 7467 ts_tablet_manager.cc:1434] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.191s user 0.007s sys 0.000s
I20260317 21:19:24.168272 7462 tablet_bootstrap.cc:492] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:24.174912 7462 tablet_bootstrap.cc:654] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:24.180835 7264 leader_election.cc:343] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.181245 7264 leader_election.cc:304] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:24.186306 8666 raft_consensus.cc:2749] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:24.200857 8624 leader_election.cc:290] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:24.210340 8718 raft_consensus.cc:493] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.210808 8718 raft_consensus.cc:515] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.213645 8718 leader_election.cc:290] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.187311 8704 consensus_queue.cc:1048] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:24.228291 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "932a5eb6cc2848f6a7d7c02ea47e5100" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:24.254588 7261 leader_election.cc:343] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.262446 7462 tablet_bootstrap.cc:492] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:24.262890 7462 ts_tablet_manager.cc:1403] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.095s user 0.008s sys 0.002s
I20260317 21:19:24.265131 7462 raft_consensus.cc:359] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.269690 8738 raft_consensus.cc:493] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.270177 8738 raft_consensus.cc:515] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.271869 8738 leader_election.cc:290] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:24.292994 7462 raft_consensus.cc:385] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.293316 7462 raft_consensus.cc:740] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.294127 7462 consensus_queue.cc:260] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.296339 7462 ts_tablet_manager.cc:1434] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260317 21:19:24.301941 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3beb9232d74e45ab713ae8241eec4c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.304180 7181 raft_consensus.cc:2468] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:24.304420 7467 tablet_bootstrap.cc:492] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:24.305435 6971 leader_election.cc:304] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:24.306080 8738 raft_consensus.cc:2804] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:24.306253 7459 tablet_bootstrap.cc:492] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:24.306391 8738 raft_consensus.cc:493] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:24.306663 7459 ts_tablet_manager.cc:1403] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.377s user 0.014s sys 0.002s
I20260317 21:19:24.306725 8738 raft_consensus.cc:3060] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.312448 8666 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.312969 8666 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.314431 8738 raft_consensus.cc:515] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.314857 8666 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.297281 7462 tablet_bootstrap.cc:492] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:24.316951 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.317556 7181 raft_consensus.cc:2468] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:24.318478 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3beb9232d74e45ab713ae8241eec4c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:24.318943 7182 raft_consensus.cc:3060] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.321368 7261 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:24.323702 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.324477 8718 raft_consensus.cc:2804] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 21:19:24.324847 8718 raft_consensus.cc:493] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:24.325165 8718 raft_consensus.cc:3060] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:24.325403 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.325987 7035 raft_consensus.cc:2468] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:24.328467 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "932a5eb6cc2848f6a7d7c02ea47e5100" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.329146 8666 raft_consensus.cc:493] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.329695 8666 raft_consensus.cc:515] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.331054 8738 leader_election.cc:290] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:24.331158 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.331759 7035 raft_consensus.cc:2468] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:24.335168 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:24.335757 7329 raft_consensus.cc:3060] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.335927 7462 tablet_bootstrap.cc:654] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:24.345180 7328 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3beb9232d74e45ab713ae8241eec4c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:24.345835 7328 raft_consensus.cc:2468] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:24.350782 7459 raft_consensus.cc:359] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.351471 7459 raft_consensus.cc:385] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.351764 7459 raft_consensus.cc:740] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.358525 7462 tablet_bootstrap.cc:492] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:24.358992 7462 ts_tablet_manager.cc:1403] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.062s user 0.007s sys 0.002s
I20260317 21:19:24.360515 8706 consensus_queue.cc:1048] T bbc7e853c2474ba9bc6981601fdc820f P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:24.366255 7459 consensus_queue.cc:260] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:24.369627 7264 leader_election.cc:343] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.370093 7264 leader_election.cc:304] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:24.361728 7462 raft_consensus.cc:359] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.372833 7462 raft_consensus.cc:385] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.373127 7462 raft_consensus.cc:740] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.375015 7462 consensus_queue.cc:260] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.380864 8666 leader_election.cc:290] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.382225 8666 raft_consensus.cc:2749] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:24.384141 7462 ts_tablet_manager.cc:1434] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.025s user 0.004s sys 0.004s
I20260317 21:19:24.385219 7462 tablet_bootstrap.cc:492] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:24.385666 7467 tablet_bootstrap.cc:654] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:24.386983 8718 raft_consensus.cc:515] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.389670 7459 ts_tablet_manager.cc:1434] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.083s user 0.007s sys 0.000s
I20260317 21:19:24.399143 6742 catalog_manager.cc:5763] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199) does not have the latest schema for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2]). Expected version 1 got 0
W20260317 21:19:24.434101 6973 consensus_peers.cc:597] T 57e721ac4b24438b9b2a0e2569027236 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 57e721ac4b24438b9b2a0e2569027236. This is attempt 6: this message will repeat every 5th retry.
I20260317 21:19:24.442988 7459 tablet_bootstrap.cc:492] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:24.446420 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 21e607674c734eaea207ed454097791d (table default.Table_5704b94c19d8479db890552467232ffb [id=30428cf56ef945279c85adbb9a300bfc], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 21e607674c734eaea207ed454097791d consensus state has no leader
I20260317 21:19:24.451714 7182 raft_consensus.cc:2468] T 1c3beb9232d74e45ab713ae8241eec4c P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:24.388743 7329 raft_consensus.cc:2468] T d4afbb1ab93a44d9ab5795b72444c690 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:24.414448 7462 tablet_bootstrap.cc:654] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:24.428969 7264 leader_election.cc:304] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:24.469415 8755 raft_consensus.cc:2804] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:24.469820 8755 raft_consensus.cc:493] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:24.470145 8755 raft_consensus.cc:3060] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.461915 8718 leader_election.cc:290] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 2 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:24.474704 6742 catalog_manager.cc:5863] Failed to send AlterTable RPC for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bc2cd63423f844aea29ed3fd839f083e consensus state has no leader
I20260317 21:19:24.488402 8755 raft_consensus.cc:515] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:24.490202 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 55216us
I20260317 21:19:24.490810 6971 leader_election.cc:304] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:24.493718 8619 raft_consensus.cc:2804] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:24.494319 8619 raft_consensus.cc:697] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:24.495234 8619 consensus_queue.cc:237] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.497730 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c3beb9232d74e45ab713ae8241eec4c" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:24.498582 7329 raft_consensus.cc:3060] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:24.504976 7264 leader_election.cc:343] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.505470 7264 leader_election.cc:304] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:24.507781 8620 consensus_queue.cc:1048] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:24.513619 8666 raft_consensus.cc:2749] T 932a5eb6cc2848f6a7d7c02ea47e5100 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:24.516208 8755 leader_election.cc:290] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
I20260317 21:19:24.519469 8753 raft_consensus.cc:493] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.521531 8753 raft_consensus.cc:515] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.523537 8753 leader_election.cc:290] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.531612 8718 raft_consensus.cc:493] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.533557 8718 raft_consensus.cc:515] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.535641 8718 leader_election.cc:290] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.542178 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:24.542972 7178 raft_consensus.cc:3060] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Advancing to term 2
I20260317 21:19:24.520681 7329 raft_consensus.cc:2468] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:24.555375 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:24.556584 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054535edcd5041e9b0429c3e7248b1b7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.557380 7462 tablet_bootstrap.cc:492] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:24.557850 7462 ts_tablet_manager.cc:1403] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.173s user 0.013s sys 0.008s
I20260317 21:19:24.561067 8718 raft_consensus.cc:493] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.561652 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c9cd4f760f44c88cdd986542755443" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.561522 8718 raft_consensus.cc:515] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.563046 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef932d50d4544ad9c5ca245bc006f29" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.563736 8718 leader_election.cc:290] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979)
W20260317 21:19:24.564113 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 66570.7us
W20260317 21:19:24.569203 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 67529us
I20260317 21:19:24.572183 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9d1076839894509805be59b751aa847" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.577224 7178 raft_consensus.cc:2468] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 2.
I20260317 21:19:24.560447 7462 raft_consensus.cc:359] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.583631 7462 raft_consensus.cc:385] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.596062 7462 raft_consensus.cc:740] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.600152 7462 consensus_queue.cc:260] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:24.609073 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 44188.9us
W20260317 21:19:24.611228 6736 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bc2cd63423f844aea29ed3fd839f083e consensus state has no leader
I20260317 21:19:24.614821 6973 leader_election.cc:304] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:24.625304 8625 raft_consensus.cc:2804] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:24.625788 8625 raft_consensus.cc:697] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:24.626948 8625 consensus_queue.cc:237] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.534941 7467 tablet_bootstrap.cc:492] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
W20260317 21:19:24.639268 7261 leader_election.cc:343] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:24.640712 7261 leader_election.cc:343] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:24.641947 7261 leader_election.cc:343] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:24.643208 7261 leader_election.cc:343] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.603505 8718 raft_consensus.cc:493] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:19:24.646696 7261 leader_election.cc:343] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.647737 8718 raft_consensus.cc:515] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.654487 7467 ts_tablet_manager.cc:1403] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.394s user 0.012s sys 0.002s
I20260317 21:19:24.655810 7084 heartbeater.cc:499] Master 127.6.122.125:32779 was elected leader, sending a full tablet report...
I20260317 21:19:24.656584 8674 consensus_queue.cc:1048] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:24.665429 8772 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:24.666009 8772 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:24.684158 8770 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:24.683987 7467 raft_consensus.cc:359] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.684842 8770 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:24.618692 7459 tablet_bootstrap.cc:654] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:24.694222 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:24.694808 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 31726fd7f40442e49b2fe0f880793206 in term 8.
I20260317 21:19:24.696082 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206, 47e1c983d0224428a9a7b147a0e5e4ef; no voters:
I20260317 21:19:24.696323 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:24.701784 8772 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:24.703630 8772 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Leader pre-election won for term 9
I20260317 21:19:24.704530 8772 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Starting leader election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:24.705461 8772 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 8 FOLLOWER]: Advancing to term 9
W20260317 21:19:24.707788 6973 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 53171.5us
I20260317 21:19:24.713959 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:24.716852 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 9 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:24.719489 8770 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 9 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:24.714068 7035 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "054535edcd5041e9b0429c3e7248b1b7" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.723950 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "cef932d50d4544ad9c5ca245bc006f29" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.724591 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:24.725163 7036 raft_consensus.cc:3060] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.726783 7038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4d64c48641b484ab443380808544d2f" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:24.727281 7038 raft_consensus.cc:3060] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Advancing to term 2
W20260317 21:19:24.727622 6971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 136180us
I20260317 21:19:24.732179 8772 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:24.742184 6683 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 47e1c983d0224428a9a7b147a0e5e4ef in current term 9: Already voted for candidate 31726fd7f40442e49b2fe0f880793206 in this term.
I20260317 21:19:24.743762 7032 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8c9cd4f760f44c88cdd986542755443" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.746876 8772 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 election: Requested vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:24.746479 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 9 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:24.750360 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291"
I20260317 21:19:24.747993 7917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 9 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
I20260317 21:19:24.752946 6661 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 9 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:24.754233 8772 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Leader election lost for term 9. Reason: could not achieve majority
I20260317 21:19:24.744948 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b9d1076839894509805be59b751aa847" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.760303 8624 raft_consensus.cc:493] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.761619 8624 raft_consensus.cc:515] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.763697 8624 leader_election.cc:290] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:24.787873 7036 raft_consensus.cc:2468] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:24.826100 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 1c3beb9232d74e45ab713ae8241eec4c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:03 UTC) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:24.827092 7007 tablet_service.cc:1558] Processing DeleteTablet for tablet 8744f6237e154dd99bc4c7e54822f7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:24.651588 8755 raft_consensus.cc:493] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.829478 8755 raft_consensus.cc:515] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.831555 8755 leader_election.cc:290] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.651707 7261 leader_election.cc:304] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
W20260317 21:19:24.832217 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 180573us
I20260317 21:19:24.691725 8753 raft_consensus.cc:493] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.835373 7013 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:24.835448 8753 raft_consensus.cc:515] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.836179 7009 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fc38947219047b5abdf71af5554a987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:24.837422 7038 raft_consensus.cc:2468] T a4d64c48641b484ab443380808544d2f P d3cb3b49e3e94537a6e434ea4e037c3f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 2.
I20260317 21:19:24.731055 7375 heartbeater.cc:499] Master 127.6.122.125:32779 was elected leader, sending a full tablet report...
I20260317 21:19:24.737044 8666 raft_consensus.cc:493] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.845008 8666 raft_consensus.cc:515] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.846827 7459 tablet_bootstrap.cc:492] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:24.847349 7459 ts_tablet_manager.cc:1403] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.405s user 0.014s sys 0.004s
I20260317 21:19:24.847796 8666 leader_election.cc:290] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.848277 8666 raft_consensus.cc:2804] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 2 FOLLOWER]: Leader election won for term 2
W20260317 21:19:24.824319 7264 leader_election.cc:343] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.848785 7264 leader_election.cc:304] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:24.858681 7264 leader_election.cc:343] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:24.859154 7264 leader_election.cc:304] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:24.856757 8753 leader_election.cc:290] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.862161 8755 raft_consensus.cc:2749] T cef932d50d4544ad9c5ca245bc006f29 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:24.862857 8754 raft_consensus.cc:2749] T 054535edcd5041e9b0429c3e7248b1b7 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:24.854805 8666 raft_consensus.cc:697] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [term 2 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:24.864312 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a44a297079347bab6f8be6abe332bfa" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:24.864075 8666 consensus_queue.cc:237] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.864938 7320 raft_consensus.cc:2468] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:24.869297 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4737071c2aec4449a05bead6af7ad32b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.870919 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:24.877043 8780 tablet_replica.cc:333] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:24.879789 8754 raft_consensus.cc:493] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.880291 8754 raft_consensus.cc:515] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.885082 8755 raft_consensus.cc:493] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:24.884552 8754 leader_election.cc:290] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.885646 8755 raft_consensus.cc:515] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.886098 7459 raft_consensus.cc:359] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.886787 7459 raft_consensus.cc:385] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.887048 7459 raft_consensus.cc:740] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.888576 7459 consensus_queue.cc:260] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.888561 8755 leader_election.cc:290] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.890597 7459 ts_tablet_manager.cc:1434] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.043s user 0.007s sys 0.000s
I20260317 21:19:24.891765 7459 tablet_bootstrap.cc:492] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:24.913655 6973 leader_election.cc:304] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:24.915371 8738 raft_consensus.cc:2804] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:24.915570 6749 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Sending DeleteTablet for 3 replicas of tablet 8a44a297079347bab6f8be6abe332bfa
I20260317 21:19:24.915834 8738 raft_consensus.cc:493] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:24.916146 8738 raft_consensus.cc:3060] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:24.917227 6749 catalog_manager.cc:6741] ReplaceTablet: tablet 8a44a297079347bab6f8be6abe332bfa deleted and replaced by tablet 19d415c5f9cd4d01a9170ae391e22d70
I20260317 21:19:24.931290 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a44a297079347bab6f8be6abe332bfa with delete_type TABLET_DATA_DELETED (replaced by tablet 19d415c5f9cd4d01a9170ae391e22d70) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:24.945967 8718 leader_election.cc:290] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:24.811077 7230 heartbeater.cc:499] Master 127.6.122.125:32779 was elected leader, sending a full tablet report...
I20260317 21:19:24.830739 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a44a297079347bab6f8be6abe332bfa" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:24.839063 8706 consensus_queue.cc:1048] T a5207adefc074c229e4020f5c9406c59 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:24.874003 8620 consensus_queue.cc:1048] T f427b15a7b2a47e18271e8884c637f4e P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:24.934594 7151 tablet_service.cc:1558] Processing DeleteTablet for tablet 3d40549347f8463797d6791136883371 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:00 UTC) from {username='slave'} at 127.0.0.1:48088
I20260317 21:19:24.935688 7150 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f34026fd2f4575bb86a9dadb733fe0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:05 UTC) from {username='slave'} at 127.0.0.1:48088
I20260317 21:19:24.936286 7156 tablet_service.cc:1558] Processing DeleteTablet for tablet fc8567ae10464694a1128ba2eb1d8ee1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:18:59 UTC) from {username='slave'} at 127.0.0.1:48088
I20260317 21:19:24.936908 7162 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a44a297079347bab6f8be6abe332bfa with delete_type TABLET_DATA_DELETED (replaced by tablet 19d415c5f9cd4d01a9170ae391e22d70) from {username='slave'} at 127.0.0.1:48088
W20260317 21:19:24.950867 6971 leader_election.cc:343] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:24.940164 7120 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 64889us
I20260317 21:19:24.962381 7467 raft_consensus.cc:385] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:24.962815 7467 raft_consensus.cc:740] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:24.963451 7467 consensus_queue.cc:260] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:24.972257 8738 raft_consensus.cc:515] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.000425 8738 leader_election.cc:290] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:25.002152 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 143498us
W20260317 21:19:25.003099 7264 leader_election.cc:343] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.003984 7264 leader_election.cc:304] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:25.005465 7264 leader_election.cc:343] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.039537 7459 tablet_bootstrap.cc:654] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.050158 8780 raft_consensus.cc:2243] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:25.050974 8780 pending_rounds.cc:70] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Trying to abort 1 pending ops.
I20260317 21:19:25.051219 8780 pending_rounds.cc:77] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412564985245696 op_type: NO_OP noop_request { }
I20260317 21:19:25.051604 8780 raft_consensus.cc:2889] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:19:24.983173 7182 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a44a297079347bab6f8be6abe332bfa" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:25.051954 8780 raft_consensus.cc:2272] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.054070 7467 ts_tablet_manager.cc:1434] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.373s user 0.004s sys 0.002s
W20260317 21:19:25.054754 6971 leader_election.cc:343] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.054953 7467 tablet_bootstrap.cc:492] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:25.056164 8780 ts_tablet_manager.cc:1916] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.059144 6683 raft_consensus.cc:1240] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Rejecting Update request from peer 21186091f2c9410ea6e585c5ef2c6291 for earlier term 8. Current term is 9. Ops: [8.425-8.429]
I20260317 21:19:25.007059 8786 tablet_replica.cc:333] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:25.026242 7181 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4737071c2aec4449a05bead6af7ad32b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.026584 7180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.007591 8718 raft_consensus.cc:2749] T b8c9cd4f760f44c88cdd986542755443 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.022480 7462 ts_tablet_manager.cc:1434] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.464s user 0.008s sys 0.000s
I20260317 21:19:25.064831 7264 leader_election.cc:304] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:25.065275 7462 ts_tablet_manager.cc:1467] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: Tablet startup took 507ms
W20260317 21:19:25.066092 7462 ts_tablet_manager.cc:1469] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4: Trace:
0317 21:19:07.859302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:07.859476 (+ 174us) service_pool.cc:225] Handling call
0317 21:19:07.860891 (+ 1415us) ts_tablet_manager.cc:669] Acquired tablet manager lock
0317 21:19:07.860982 (+ 91us) ts_tablet_manager.cc:682] Creating new metadata...
0317 21:19:07.870901 (+ 9919us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:07.887816 (+ 16915us) tablet_replica.cc:206] Creating consensus instance
0317 21:19:07.888223 (+ 407us) inbound_call.cc:173] Queueing success response
0317 21:19:24.384863 (+16496640us) ts_tablet_manager.cc:1379] Bootstrapping tablet
0317 21:19:24.558109 (+173246us) ts_tablet_manager.cc:1435] Starting tablet replica
0317 21:19:24.558644 (+ 535us) tablet_replica.cc:261] Starting instrumentation
0317 21:19:24.559114 (+ 470us) tablet_replica.cc:280] Starting consensus
Metrics: {"raft.queue_time_us":1900,"raft.run_cpu_time_us":199,"raft.run_wall_time_us":201,"tablet-open.queue_time_us":16496749}
I20260317 21:19:25.068336 7462 tablet_bootstrap.cc:492] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:25.058606 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8a44a297079347bab6f8be6abe332bfa" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:25.065702 8718 raft_consensus.cc:2749] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.068857 7322 raft_consensus.cc:3060] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:25.074376 8774 consensus_queue.cc:1059] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 }, Status: INVALID_TERM, Last received: 8.424, Next index: 425, Last known committed idx: 423, Time since last communication: 0.001s
I20260317 21:19:25.076306 8756 raft_consensus.cc:3055] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 8 LEADER]: Stepping down as leader of term 8
I20260317 21:19:25.076853 8756 raft_consensus.cc:740] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 8 LEADER]: Becoming Follower/Learner. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:25.077620 8756 consensus_queue.cc:260] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 8.429, Last appended by leader: 429, Current term: 8, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:25.078914 8756 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:19:25.084920 8770 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 8 FOLLOWER]: Advancing to term 9
I20260317 21:19:25.094025 8770 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 9 FOLLOWER]: Leader pre-election lost for term 9. Reason: could not achieve majority
I20260317 21:19:25.104111 7459 tablet_bootstrap.cc:492] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:25.030372 7179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.030735 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f69403b3b811419e87916a381656e1ec" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.031914 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2607b9978c32466eb6dc8b8acfb429ce" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.034466 7177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0dd4c1efc0624957832c17dc7a4db3f3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.105068 7459 ts_tablet_manager.cc:1403] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.213s user 0.004s sys 0.008s
I20260317 21:19:25.118189 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet b9d1076839894509805be59b751aa847 with delete_type TABLET_DATA_DELETED (replaced by tablet d2aba7eba6df430e86e003ef3a179912) from {username='slave'} at 127.0.0.1:33804
I20260317 21:19:25.119621 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 8744f6237e154dd99bc4c7e54822f7be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:06 UTC) from {username='slave'} at 127.0.0.1:33804
I20260317 21:19:25.120110 7459 raft_consensus.cc:359] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.120872 7459 raft_consensus.cc:385] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.120931 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 7fc38947219047b5abdf71af5554a987 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:02 UTC) from {username='slave'} at 127.0.0.1:33804
I20260317 21:19:25.121131 7459 raft_consensus.cc:740] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.122084 7306 tablet_service.cc:1558] Processing DeleteTablet for tablet 8a44a297079347bab6f8be6abe332bfa with delete_type TABLET_DATA_DELETED (replaced by tablet 19d415c5f9cd4d01a9170ae391e22d70) from {username='slave'} at 127.0.0.1:33804
I20260317 21:19:25.121738 7459 consensus_queue.cc:260] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.129868 7459 ts_tablet_manager.cc:1434] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.024s user 0.005s sys 0.000s
I20260317 21:19:25.130863 7459 tablet_bootstrap.cc:492] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:25.137229 8767 consensus_queue.cc:1048] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:25.146795 7459 tablet_bootstrap.cc:654] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.153003 8781 raft_consensus.cc:493] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.153497 8781 raft_consensus.cc:515] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.155550 8781 leader_election.cc:290] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:25.065640 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 60204.8us
I20260317 21:19:25.161155 7264 leader_election.cc:304] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:25.162073 8781 raft_consensus.cc:2804] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:25.162580 8781 raft_consensus.cc:697] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:25.163468 8781 consensus_queue.cc:237] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.166491 7459 tablet_bootstrap.cc:492] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:25.166903 7459 ts_tablet_manager.cc:1403] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.036s user 0.009s sys 0.003s
W20260317 21:19:25.167199 7261 leader_election.cc:343] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.169719 7459 raft_consensus.cc:359] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.170399 7459 raft_consensus.cc:385] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.170662 7459 raft_consensus.cc:740] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:25.171165 7261 leader_election.cc:343] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.171355 7459 consensus_queue.cc:260] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.173991 7459 ts_tablet_manager.cc:1434] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.007s user 0.003s sys 0.002s
W20260317 21:19:25.179055 7261 leader_election.cc:343] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:25.179725 7261 leader_election.cc:343] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.179807 8782 raft_consensus.cc:493] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.180398 8782 raft_consensus.cc:515] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.183247 8782 leader_election.cc:290] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:25.185132 7261 leader_election.cc:343] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:25.193600 7261 leader_election.cc:343] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.194279 8792 tablet_replica.cc:333] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:25.195554 8792 raft_consensus.cc:2243] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:25.196139 8792 raft_consensus.cc:2272] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:24.812143 7119 consensus_peers.cc:597] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fc8567ae10464694a1128ba2eb1d8ee1. This is attempt 6: this message will repeat every 5th retry.
I20260317 21:19:25.142215 7467 tablet_bootstrap.cc:654] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.165987 8786 raft_consensus.cc:2243] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:25.202708 8786 raft_consensus.cc:2272] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.210996 7459 tablet_bootstrap.cc:492] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:25.221243 7462 tablet_bootstrap.cc:654] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:25.203390 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 391522us
I20260317 21:19:25.208108 7467 tablet_bootstrap.cc:492] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:25.232167 7467 ts_tablet_manager.cc:1403] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.177s user 0.010s sys 0.005s
I20260317 21:19:25.190954 7322 raft_consensus.cc:2468] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:25.212761 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d51e3d4702724484b776a95eb5acf7ef" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.220065 7177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfff4c8478b64436a666a14c133c36d3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:25.235739 7261 leader_election.cc:343] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:25.192462 7264 leader_election.cc:343] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:25.236901 7261 leader_election.cc:343] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.247985 6973 leader_election.cc:304] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:25.249023 8625 raft_consensus.cc:2804] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:25.249504 8625 raft_consensus.cc:697] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:25.250413 8625 consensus_queue.cc:237] T 8a44a297079347bab6f8be6abe332bfa P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.263665 8780 ts_tablet_manager.cc:1929] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:25.264129 8780 log.cc:1199] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/1c3beb9232d74e45ab713ae8241eec4c
I20260317 21:19:25.216575 7459 tablet_bootstrap.cc:654] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.277778 8738 consensus_queue.cc:1048] T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:25.285260 8780 ts_tablet_manager.cc:1950] T 1c3beb9232d74e45ab713ae8241eec4c P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:25.294005 6737 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 1c3beb9232d74e45ab713ae8241eec4c (table default.Table_bdd5608a590847cf98a367c051f62b45 [id=70d2f8f2445b4fab82ec7794c49c553f]) successfully deleted
I20260317 21:19:25.236824 7264 leader_election.cc:304] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:25.298233 7467 raft_consensus.cc:359] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.298929 7467 raft_consensus.cc:385] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.299192 7467 raft_consensus.cc:740] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
W20260317 21:19:25.299252 7119 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 65027.5us
I20260317 21:19:25.299785 7467 consensus_queue.cc:260] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.301963 7467 ts_tablet_manager.cc:1434] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.069s user 0.002s sys 0.004s
I20260317 21:19:25.307951 8786 ts_tablet_manager.cc:1916] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.314540 7459 tablet_bootstrap.cc:492] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:25.314972 7459 ts_tablet_manager.cc:1403] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.104s user 0.011s sys 0.000s
I20260317 21:19:25.318185 8767 raft_consensus.cc:493] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.312011 8666 raft_consensus.cc:493] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.318635 8767 raft_consensus.cc:515] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.319255 8666 raft_consensus.cc:515] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.320498 8767 leader_election.cc:290] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:25.326201 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 133747us
W20260317 21:19:25.327441 7264 leader_election.cc:343] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.308905 8781 raft_consensus.cc:493] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.330242 8666 leader_election.cc:290] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.317432 7459 raft_consensus.cc:359] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.344991 7459 raft_consensus.cc:385] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.345295 7459 raft_consensus.cc:740] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.330536 8755 raft_consensus.cc:2749] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.320199 8718 raft_consensus.cc:493] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.332438 7319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:25.334354 8792 ts_tablet_manager.cc:1916] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.341709 7264 leader_election.cc:304] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:25.322834 7177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.349422 7177 raft_consensus.cc:2468] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:25.332986 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d69e14125d7d4412869c20759acc2336" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:25.351233 7261 leader_election.cc:343] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.351965 8755 raft_consensus.cc:493] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.348004 8718 raft_consensus.cc:515] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.352878 6971 leader_election.cc:304] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:25.352984 8666 raft_consensus.cc:2749] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.352767 8755 raft_consensus.cc:515] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.353724 8767 raft_consensus.cc:2804] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:25.354053 8767 raft_consensus.cc:493] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:25.354405 8767 raft_consensus.cc:3060] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:25.354681 8755 leader_election.cc:290] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.354866 8718 leader_election.cc:290] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.347321 8781 raft_consensus.cc:515] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.357311 8781 leader_election.cc:290] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.357846 7319 raft_consensus.cc:2468] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:25.360503 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.360831 7177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae1b2a549af6485ca3faee3f0ad845a2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.361112 7183 raft_consensus.cc:2468] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:25.360527 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4af622a9190148ee80b2d5d3d7a92a20" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:25.362047 7459 consensus_queue.cc:260] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:25.363113 7261 leader_election.cc:343] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.363711 7261 leader_election.cc:304] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:25.364324 7459 ts_tablet_manager.cc:1434] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.049s user 0.007s sys 0.001s
I20260317 21:19:25.364447 8781 raft_consensus.cc:2804] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:25.364885 8781 raft_consensus.cc:493] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:25.365216 8781 raft_consensus.cc:3060] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:25.369870 8767 raft_consensus.cc:515] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.371994 8803 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:19:25.372167 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 44759us
W20260317 21:19:25.364462 7261 leader_election.cc:343] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.375247 8767 leader_election.cc:290] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
W20260317 21:19:25.375541 8795 log.cc:927] Time spent T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: Append to log took a long time: real 0.139s user 0.001s sys 0.000s
I20260317 21:19:25.377544 7319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:25.378823 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:25.379366 7184 raft_consensus.cc:3060] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:25.374042 8803 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:25.385310 7467 tablet_bootstrap.cc:492] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:25.377135 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.385850 7036 raft_consensus.cc:2468] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:25.388268 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 10 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:25.389276 6751 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31726fd7f40442e49b2fe0f880793206 for term 10 because replica has last-logged OpId of term: 8 index: 429, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 424.
I20260317 21:19:25.393191 8803 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:25.394814 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 10 candidate_status { last_received { term: 8 index: 424 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:25.395253 7467 tablet_bootstrap.cc:654] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.395655 7918 raft_consensus.cc:2410] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 9 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 31726fd7f40442e49b2fe0f880793206 for term 10 because replica has last-logged OpId of term: 8 index: 429, which is greater than that of the candidate, which has last-logged OpId of term: 8 index: 424.
I20260317 21:19:25.396929 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:25.408664 7462 tablet_bootstrap.cc:492] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:25.409098 7462 ts_tablet_manager.cc:1403] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.341s user 0.019s sys 0.003s
I20260317 21:19:25.409943 8780 tablet_replica.cc:333] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:25.410681 7459 tablet_bootstrap.cc:492] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:25.411375 7462 raft_consensus.cc:359] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.412290 7462 raft_consensus.cc:385] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.414040 8780 raft_consensus.cc:2243] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:25.415618 8780 raft_consensus.cc:2272] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.405144 8803 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Leader pre-election lost for term 10. Reason: could not achieve majority
I20260317 21:19:25.417955 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f69403b3b811419e87916a381656e1ec" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.417980 8780 ts_tablet_manager.cc:1916] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.431211 7459 tablet_bootstrap.cc:654] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.432714 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2607b9978c32466eb6dc8b8acfb429ce" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.433435 7036 raft_consensus.cc:2468] T 2607b9978c32466eb6dc8b8acfb429ce P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:25.433728 8781 raft_consensus.cc:515] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.435050 7319 raft_consensus.cc:2393] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d3cb3b49e3e94537a6e434ea4e037c3f in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:25.440593 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0dd4c1efc0624957832c17dc7a4db3f3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.442299 7184 raft_consensus.cc:2468] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:25.442828 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d51e3d4702724484b776a95eb5acf7ef" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.444208 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfff4c8478b64436a666a14c133c36d3" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.445446 6971 leader_election.cc:304] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: 2f539d1a4a584b8ca83928be49369bf4
I20260317 21:19:25.446524 8738 raft_consensus.cc:2804] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:25.446973 8738 raft_consensus.cc:697] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:25.448896 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae1b2a549af6485ca3faee3f0ad845a2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.450454 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4af622a9190148ee80b2d5d3d7a92a20" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.447798 8738 consensus_queue.cc:237] T 788f0ee1f7de47a49c5601c9b1494885 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.451642 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.445562 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d69e14125d7d4412869c20759acc2336" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.455344 8756 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 9 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.455854 8756 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 9 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:25.458984 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 10 candidate_status { last_received { term: 8 index: 429 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:25.459625 6683 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 9.
I20260317 21:19:25.461036 6736 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 10 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206; no voters:
I20260317 21:19:25.461977 8774 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 9 FOLLOWER]: Leader pre-election won for term 10
I20260317 21:19:25.462332 8774 raft_consensus.cc:493] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 9 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:25.462667 8774 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:19:25.466490 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 10 candidate_status { last_received { term: 8 index: 429 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:25.467083 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 9 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 9.
I20260317 21:19:25.468470 8756 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 10 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:25.478204 7467 tablet_bootstrap.cc:492] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:25.478777 7467 ts_tablet_manager.cc:1403] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.094s user 0.011s sys 0.000s
W20260317 21:19:25.481895 6971 consensus_peers.cc:597] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Couldn't send request to peer c0ef92fef382401a9564c17e0c223f6a. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:25.486752 8774 raft_consensus.cc:515] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 10 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:25.489882 6683 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 10 candidate_status { last_received { term: 8 index: 429 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206"
I20260317 21:19:25.490558 6683 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:19:25.491551 7459 tablet_bootstrap.cc:492] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:25.491976 7459 ts_tablet_manager.cc:1403] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.081s user 0.007s sys 0.010s
I20260317 21:19:25.494834 7459 raft_consensus.cc:359] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.495519 7459 raft_consensus.cc:385] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.495785 7459 raft_consensus.cc:740] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.499477 6683 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 10.
I20260317 21:19:25.496384 7459 consensus_queue.cc:260] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.501009 6736 leader_election.cc:304] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 10 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206; no voters:
I20260317 21:19:25.502110 8756 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 10 FOLLOWER]: Leader election won for term 10
I20260317 21:19:25.502981 7459 ts_tablet_manager.cc:1434] T 4737071c2aec4449a05bead6af7ad32b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.011s user 0.003s sys 0.004s
I20260317 21:19:25.412566 7462 raft_consensus.cc:740] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.503871 7459 tablet_bootstrap.cc:492] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
W20260317 21:19:25.507270 8804 log.cc:927] Time spent T 70560dde990843e0b7c74ae3206c58e4 P d3cb3b49e3e94537a6e434ea4e037c3f: Append to log took a long time: real 0.077s user 0.001s sys 0.000s
I20260317 21:19:25.509088 8781 leader_election.cc:290] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.513556 7467 raft_consensus.cc:359] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.518879 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "21186091f2c9410ea6e585c5ef2c6291" candidate_term: 10 candidate_status { last_received { term: 8 index: 429 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef"
I20260317 21:19:25.519381 7918 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 9 FOLLOWER]: Advancing to term 10
I20260317 21:19:25.527361 7918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 21186091f2c9410ea6e585c5ef2c6291 in term 10.
I20260317 21:19:25.528236 7459 tablet_bootstrap.cc:654] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.530670 8774 leader_election.cc:290] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [CANDIDATE]: Term 10 election: Requested vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:25.517203 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:25.537487 7467 raft_consensus.cc:385] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.537636 7183 raft_consensus.cc:2393] T 788f0ee1f7de47a49c5601c9b1494885 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 2f539d1a4a584b8ca83928be49369bf4 in current term 1: Already voted for candidate d3cb3b49e3e94537a6e434ea4e037c3f in this term.
W20260317 21:19:25.530193 8801 log.cc:927] Time spent T 70560dde990843e0b7c74ae3206c58e4 P c0ef92fef382401a9564c17e0c223f6a: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260317 21:19:25.541742 7467 raft_consensus.cc:740] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.542882 7467 consensus_queue.cc:260] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.503882 7462 consensus_queue.cc:260] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.546489 7462 ts_tablet_manager.cc:1434] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.137s user 0.005s sys 0.000s
I20260317 21:19:25.547576 7462 tablet_bootstrap.cc:492] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:25.558106 7467 ts_tablet_manager.cc:1434] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.079s user 0.001s sys 0.006s
I20260317 21:19:25.558547 8756 raft_consensus.cc:697] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [term 10 LEADER]: Becoming Leader. State: Replica: 21186091f2c9410ea6e585c5ef2c6291, State: Running, Role: LEADER
I20260317 21:19:25.559494 8756 consensus_queue.cc:237] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 424, Committed index: 424, Last appended: 8.429, Last appended by leader: 429, Current term: 10, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:25.563868 8774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:25.568817 8774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:25.540349 8792 ts_tablet_manager.cc:1929] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:25.569588 8792 log.cc:1199] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/b9d1076839894509805be59b751aa847
I20260317 21:19:25.571271 8792 ts_tablet_manager.cc:1950] T b9d1076839894509805be59b751aa847 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:25.571425 8780 ts_tablet_manager.cc:1929] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:25.574394 6736 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet b9d1076839894509805be59b751aa847 (table default.Table_2e5acf8292c248929fd6beead0350ea4 [id=a65612697cf24d359b652484c8338915]) successfully deleted
W20260317 21:19:25.580444 6660 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet 163cc8c04ca94a269c7ae2efaa162525 (table default.Table_32efbd6caca74d13a6d567c3f470d9a1 [id=0be06b86c4114f8598f6a1bfa74e3cfa], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 163cc8c04ca94a269c7ae2efaa162525 consensus state has no leader
I20260317 21:19:25.580920 7462 tablet_bootstrap.cc:654] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.584759 7467 tablet_bootstrap.cc:492] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:25.597085 8792 tablet_replica.cc:333] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:25.598161 8792 raft_consensus.cc:2243] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:25.598800 8792 raft_consensus.cc:2272] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.601014 8792 ts_tablet_manager.cc:1916] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.615335 7462 tablet_bootstrap.cc:492] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:25.616277 7462 ts_tablet_manager.cc:1403] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.008s
I20260317 21:19:25.617025 7467 tablet_bootstrap.cc:654] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.625283 8783 raft_consensus.cc:493] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.625813 8783 raft_consensus.cc:515] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.630589 8783 leader_election.cc:290] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:25.628710 7462 raft_consensus.cc:359] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.645212 7462 raft_consensus.cc:385] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.645495 7462 raft_consensus.cc:740] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.646055 7462 consensus_queue.cc:260] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.653906 7462 ts_tablet_manager.cc:1434] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260317 21:19:25.655016 7462 tablet_bootstrap.cc:492] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:25.658111 7467 tablet_bootstrap.cc:492] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:25.658522 7467 ts_tablet_manager.cc:1403] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.000s
I20260317 21:19:25.682178 7467 raft_consensus.cc:359] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.682888 7467 raft_consensus.cc:385] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.683153 7467 raft_consensus.cc:740] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.683769 7467 consensus_queue.cc:260] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:25.689452 7264 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.6.122.124:38857, user_credentials={real_user=slave}} blocked reactor thread for 48090.6us
I20260317 21:19:25.690582 7264 leader_election.cc:304] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:25.701344 7467 ts_tablet_manager.cc:1434] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.043s user 0.003s sys 0.004s
I20260317 21:19:25.702279 7467 tablet_bootstrap.cc:492] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:25.709465 8781 raft_consensus.cc:2804] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:25.710397 8781 raft_consensus.cc:493] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:25.710721 8781 raft_consensus.cc:3060] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:25.721632 8781 raft_consensus.cc:515] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.725697 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
W20260317 21:19:25.729321 7261 leader_election.cc:343] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.730544 8781 leader_election.cc:290] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:25.735883 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 45378.2us
I20260317 21:19:25.736752 7462 tablet_bootstrap.cc:654] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.611156 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "788f0ee1f7de47a49c5601c9b1494885" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:25.636251 7459 tablet_bootstrap.cc:492] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:25.674093 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:25.742406 7459 ts_tablet_manager.cc:1403] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.239s user 0.008s sys 0.016s
I20260317 21:19:25.742504 7036 raft_consensus.cc:2393] T bc2cd63423f844aea29ed3fd839f083e P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:25.746104 7459 raft_consensus.cc:359] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.747025 7459 raft_consensus.cc:385] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.747329 7459 raft_consensus.cc:740] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.747962 7459 consensus_queue.cc:260] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.748916 7467 tablet_bootstrap.cc:654] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.750149 7459 ts_tablet_manager.cc:1434] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:25.751187 7459 tablet_bootstrap.cc:492] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:25.757066 8792 ts_tablet_manager.cc:1929] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:25.757817 8792 log.cc:1199] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/8744f6237e154dd99bc4c7e54822f7be
I20260317 21:19:25.759282 8792 ts_tablet_manager.cc:1950] T 8744f6237e154dd99bc4c7e54822f7be P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:25.761091 8780 log.cc:1199] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/8744f6237e154dd99bc4c7e54822f7be
I20260317 21:19:25.762092 8780 ts_tablet_manager.cc:1950] T 8744f6237e154dd99bc4c7e54822f7be P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:25.764930 6737 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 8744f6237e154dd99bc4c7e54822f7be (table default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]) successfully deleted
W20260317 21:19:25.765964 7264 leader_election.cc:343] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.766499 7264 leader_election.cc:304] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:25.769451 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 94023.7us
I20260317 21:19:25.774732 6736 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 8744f6237e154dd99bc4c7e54822f7be (table default.Table_e717b96ed78c4b5d8aab97d49666ebea [id=00866233f6514139a11f8c1e0f546f27]) successfully deleted
I20260317 21:19:25.776860 8781 raft_consensus.cc:2749] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.777586 8782 raft_consensus.cc:493] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.778002 8782 raft_consensus.cc:515] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.778421 7462 tablet_bootstrap.cc:492] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:25.778956 7462 ts_tablet_manager.cc:1403] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.124s user 0.011s sys 0.001s
I20260317 21:19:25.779992 8782 leader_election.cc:290] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.784140 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2c52f4843234e8584a66e5e8e065667" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:25.785938 7261 leader_election.cc:343] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.787984 8792 tablet_replica.cc:333] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:25.789675 8792 raft_consensus.cc:2243] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:25.788295 7462 raft_consensus.cc:359] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.797148 7462 raft_consensus.cc:385] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.797503 7462 raft_consensus.cc:740] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.798197 7462 consensus_queue.cc:260] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.799208 8786 ts_tablet_manager.cc:1929] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:25.799733 8786 log.cc:1199] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/3d40549347f8463797d6791136883371
I20260317 21:19:25.801693 7462 ts_tablet_manager.cc:1434] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.022s user 0.005s sys 0.001s
I20260317 21:19:25.802589 7462 tablet_bootstrap.cc:492] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:25.799139 8792 raft_consensus.cc:2272] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.807008 8792 ts_tablet_manager.cc:1916] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.767971 7264 leader_election.cc:304] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f; no voters: c0ef92fef382401a9564c17e0c223f6a
I20260317 21:19:25.810791 8781 raft_consensus.cc:2804] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:25.811200 8781 raft_consensus.cc:493] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:25.811519 8781 raft_consensus.cc:3060] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:25.819562 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 51670.3us
W20260317 21:19:25.822093 7264 leader_election.cc:343] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.822544 7264 leader_election.cc:304] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:25.823210 8782 raft_consensus.cc:2749] T 0dd4c1efc0624957832c17dc7a4db3f3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:25.826452 7264 leader_election.cc:343] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.826869 7264 leader_election.cc:304] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:25.828874 7462 tablet_bootstrap.cc:654] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.840353 8767 consensus_queue.cc:1048] T d4afbb1ab93a44d9ab5795b72444c690 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Connected to new peer: Peer: permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:19:25.845191 6973 consensus_peers.cc:597] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f -> Peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Couldn't send request to peer 2f539d1a4a584b8ca83928be49369bf4. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:25.872182 8792 ts_tablet_manager.cc:1929] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:25.877463 8792 log.cc:1199] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/7fc38947219047b5abdf71af5554a987
I20260317 21:19:25.879599 7467 tablet_bootstrap.cc:492] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:25.880077 7467 ts_tablet_manager.cc:1403] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.178s user 0.015s sys 0.016s
I20260317 21:19:25.881045 8792 ts_tablet_manager.cc:1950] T 7fc38947219047b5abdf71af5554a987 P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:25.882784 7459 tablet_bootstrap.cc:654] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.882499 7467 raft_consensus.cc:359] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.883304 7467 raft_consensus.cc:385] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.883574 7467 raft_consensus.cc:740] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.885017 8786 ts_tablet_manager.cc:1950] T 3d40549347f8463797d6791136883371 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:25.887953 8782 raft_consensus.cc:2749] T d51e3d4702724484b776a95eb5acf7ef P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:25.889577 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 63155.6us
W20260317 21:19:25.890384 7264 leader_election.cc:343] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.890821 7264 leader_election.cc:304] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:25.891762 8782 raft_consensus.cc:2749] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.894886 6736 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 7fc38947219047b5abdf71af5554a987 (table default.Table_ef1aaeadf5354f7e99d9db422dfc6e63 [id=46ad456ffb374be585c1a222df6dabc6]) successfully deleted
I20260317 21:19:25.819042 8781 raft_consensus.cc:515] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.899005 8780 tablet_replica.cc:333] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:25.900193 8780 raft_consensus.cc:2243] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:25.899879 8792 tablet_replica.cc:333] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: stopping tablet replica
I20260317 21:19:25.901185 8780 raft_consensus.cc:2272] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.901268 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2607b9978c32466eb6dc8b8acfb429ce" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:25.901860 7178 raft_consensus.cc:3060] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:25.903133 8780 ts_tablet_manager.cc:1916] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.904467 8792 raft_consensus.cc:2243] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:25.908574 8792 raft_consensus.cc:2272] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:25.910763 8792 ts_tablet_manager.cc:1916] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:25.910941 7178 raft_consensus.cc:2468] T 2607b9978c32466eb6dc8b8acfb429ce P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 1.
I20260317 21:19:25.912709 7261 leader_election.cc:304] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:25.884182 7467 consensus_queue.cc:260] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.919617 8818 raft_consensus.cc:2804] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:25.928973 7467 ts_tablet_manager.cc:1434] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.049s user 0.008s sys 0.000s
I20260317 21:19:25.937058 8782 raft_consensus.cc:493] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:25.937578 8782 raft_consensus.cc:515] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.939551 8782 leader_election.cc:290] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.941869 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "32be4818a67a41c998f04d7cedf7356e" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:25.946321 7261 leader_election.cc:343] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:25.950755 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 60396.2us
I20260317 21:19:25.951175 7467 tablet_bootstrap.cc:492] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:25.955156 8781 leader_election.cc:290] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:25.957794 7462 tablet_bootstrap.cc:492] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:25.958250 7467 tablet_bootstrap.cc:654] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:25.958333 7462 ts_tablet_manager.cc:1403] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.156s user 0.022s sys 0.008s
I20260317 21:19:25.960980 7462 raft_consensus.cc:359] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:25.965420 7918 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Refusing update from remote peer 21186091f2c9410ea6e585c5ef2c6291: Log matching property violated. Preceding OpId in replica: term: 8 index: 429. Preceding OpId from leader: term: 10 index: 430. (index mismatch)
I20260317 21:19:25.967584 8774 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 430, Last known committed idx: 424, Time since last communication: 0.001s
I20260317 21:19:25.973659 8786 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48088 (request call id 0) took 1094 ms. Trace:
I20260317 21:19:25.977734 8786 rpcz_store.cc:276] 0317 21:19:24.933988 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.934247 (+ 259us) service_pool.cc:225] Handling call
0317 21:19:25.006527 (+ 72280us) ts_tablet_manager.cc:1151] Deleting tablet 3d40549347f8463797d6791136883371
0317 21:19:25.006600 (+ 73us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:25.201804 (+195204us) spinlock_profiling.cc:243] Waited 2.74 ms on lock 0x7b54001234d4. stack: 00007f622fe56423 00007f622fe56320 00007f622f9cb574 0000000000509544 00000000005094d9 00007f6236eb17a1 00007f6236f50292 00007f623737257c 00007f6237fc98aa 00007f6237ffa510 00007f6237fe0add 00007f6237fe0a89 00007f6237fe0a19 00007f6237fe09e1 00007f6237fdfc0d 00000000005ab084
0317 21:19:25.538765 (+336961us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:25.798947 (+260182us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:25.973553 (+174606us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":35072,"spinlock_wait_cycles":6293248,"tablet-delete.queue_time_us":57423,"thread_start_us":58824,"threads_started":1}
I20260317 21:19:25.983955 7462 raft_consensus.cc:385] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:25.984297 7462 raft_consensus.cc:740] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:25.969946 8818 raft_consensus.cc:697] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [term 1 LEADER]: Becoming Leader. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Running, Role: LEADER
I20260317 21:19:25.985803 8818 consensus_queue.cc:237] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:25.977159 7264 leader_election.cc:343] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.990077 7264 leader_election.cc:304] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:25.991143 7264 leader_election.cc:343] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:25.991946 8782 raft_consensus.cc:2749] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:26.000720 7264 leader_election.cc:304] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:26.001428 8782 raft_consensus.cc:2749] T 4af622a9190148ee80b2d5d3d7a92a20 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:25.994575 7462 consensus_queue.cc:260] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.003466 7462 ts_tablet_manager.cc:1434] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260317 21:19:26.004899 7462 tablet_bootstrap.cc:492] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:26.011261 6683 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Refusing update from remote peer 21186091f2c9410ea6e585c5ef2c6291: Log matching property violated. Preceding OpId in replica: term: 8 index: 424. Preceding OpId from leader: term: 10 index: 430. (index mismatch)
I20260317 21:19:26.013092 8820 raft_consensus.cc:493] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.013501 8774 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [LEADER]: Connected to new peer: Peer: permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 430, Last known committed idx: 423, Time since last communication: 0.000s
I20260317 21:19:26.013587 8820 raft_consensus.cc:515] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.016348 8820 leader_election.cc:290] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:26.002986 7467 tablet_bootstrap.cc:492] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.030773 7467 ts_tablet_manager.cc:1403] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.080s user 0.010s sys 0.003s
I20260317 21:19:26.032244 6734 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 3d40549347f8463797d6791136883371 (table default.Table_1c93ac734b4d4ff68afbd9c72d470944 [id=5939f7e719474a598b8428b8af9a5b8b]) successfully deleted
I20260317 21:19:26.021047 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6cf1a0922ed54674aaff8b0f49751c13" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:26.034181 7467 raft_consensus.cc:359] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.034852 7467 raft_consensus.cc:385] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.035095 7467 raft_consensus.cc:740] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
W20260317 21:19:26.035138 7261 leader_election.cc:343] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.035656 7467 consensus_queue.cc:260] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.037730 7467 ts_tablet_manager.cc:1434] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.007s user 0.006s sys 0.000s
I20260317 21:19:26.038343 7459 tablet_bootstrap.cc:492] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f: No bootstrap required, opened a new log
I20260317 21:19:26.038846 7459 ts_tablet_manager.cc:1403] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent bootstrapping tablet: real 0.288s user 0.011s sys 0.004s
I20260317 21:19:26.040495 8780 ts_tablet_manager.cc:1929] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:26.041046 8780 log.cc:1199] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/00f34026fd2f4575bb86a9dadb733fe0
I20260317 21:19:26.041885 8780 ts_tablet_manager.cc:1950] T 00f34026fd2f4575bb86a9dadb733fe0 P d3cb3b49e3e94537a6e434ea4e037c3f: Deleting consensus metadata
I20260317 21:19:26.049818 8824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
W20260317 21:19:26.053346 7264 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 62230.7us
I20260317 21:19:26.060880 8786 tablet_replica.cc:333] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:26.061775 7467 tablet_bootstrap.cc:492] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.062786 7178 raft_consensus.cc:3060] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.063470 8786 raft_consensus.cc:2243] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:26.067751 8786 raft_consensus.cc:2272] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:26.068065 7467 tablet_bootstrap.cc:654] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.076392 8786 ts_tablet_manager.cc:1916] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:26.087193 8824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:26.041719 7459 raft_consensus.cc:359] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.076938 8738 raft_consensus.cc:493] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.090103 8738 raft_consensus.cc:515] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:26.091492 6736 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bc2cd63423f844aea29ed3fd839f083e consensus state has no leader
I20260317 21:19:26.092489 8738 leader_election.cc:290] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.093116 7459 raft_consensus.cc:385] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.093580 7459 raft_consensus.cc:740] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Initialized, Role: FOLLOWER
W20260317 21:19:26.098205 7264 leader_election.cc:343] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.103909 8780 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:34424 (request call id 3) took 1277 ms. Trace:
I20260317 21:19:26.104552 8780 rpcz_store.cc:276] 0317 21:19:24.826363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.834254 (+ 7891us) service_pool.cc:225] Handling call
0317 21:19:25.840004 (+1005750us) ts_tablet_manager.cc:1151] Deleting tablet 00f34026fd2f4575bb86a9dadb733fe0
0317 21:19:25.898868 (+ 58864us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:25.986495 (+ 87627us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.040259 (+ 53764us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.103837 (+ 63578us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12984,"tablet-delete.queue_time_us":991519}
I20260317 21:19:26.094194 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:26.106521 7183 raft_consensus.cc:2468] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:26.112267 8620 raft_consensus.cc:493] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.112313 7264 leader_election.cc:304] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:26.112912 8620 raft_consensus.cc:515] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.115013 8620 leader_election.cc:290] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.115844 7467 tablet_bootstrap.cc:492] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.116281 7467 ts_tablet_manager.cc:1403] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.055s user 0.006s sys 0.005s
I20260317 21:19:26.116885 6744 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Sending DeleteTablet for 3 replicas of tablet 4c19bac7772a498f87f48e6095310347
I20260317 21:19:26.118335 7319 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.118675 6744 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50358 (request call id 52) took 2000 ms. Trace:
I20260317 21:19:26.118979 7319 raft_consensus.cc:2468] T 16f64073188f4d06859e1a701df73242 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:26.118796 7467 raft_consensus.cc:359] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.119429 7467 raft_consensus.cc:385] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.119699 7467 raft_consensus.cc:740] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.120397 7467 consensus_queue.cc:260] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.121580 7151 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c19bac7772a498f87f48e6095310347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:24 UTC) from {username='slave'} at 127.0.0.1:48088
I20260317 21:19:26.122192 8782 raft_consensus.cc:2749] T d69e14125d7d4412869c20759acc2336 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:26.122483 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c19bac7772a498f87f48e6095310347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:24 UTC) from {username='slave'} at 127.0.0.1:33804
I20260317 21:19:26.123662 7264 leader_election.cc:304] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:26.119052 6744 rpcz_store.cc:276] 0317 21:19:24.118493 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.118653 (+ 160us) service_pool.cc:225] Handling call
0317 21:19:24.129211 (+ 10558us) spinlock_profiling.cc:243] Waited 8.22 ms on lock 0x7b5400030554. stack: 00007f5fbab7a423 00007f5fbab7a320 00007f5fba6ef574 0000000000509544 00000000005094d9 00007f5fc1bd62c0 00007f5fc1c74e6a 00007f5fc397721c 00007f5fc3a71386 00007f5fc1697024 00007f5fc1696fb2 00007f5fc1696ee1 00007f5fc1696e5c 00007f5fc1696112 00007f5fbf96834c 00007f5fbf967786
0317 21:19:24.130547 (+ 1336us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:24.131425 (+ 878us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:24.131738 (+ 313us) catalog_manager.cc:2824] Locking tablets
0317 21:19:24.132544 (+ 806us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:26.111620 (+1979076us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:19:26.111949 (+ 329us) catalog_manager.cc:2872] Committing in-memory state
0317 21:19:26.112072 (+ 123us) catalog_manager.cc:2876] Aborting table tasks
0317 21:19:26.118592 (+ 6520us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:24.185565 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:24.186038 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:24.186065 (+ 27us) write_op.cc:435] Acquired schema lock
0317 21:19:24.186077 (+ 12us) tablet.cc:661] Decoding operations
0317 21:19:24.186437 (+ 360us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:24.186476 (+ 39us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:24.186534 (+ 58us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:24.186825 (+ 291us) tablet.cc:700] Row locks acquired
0317 21:19:24.186843 (+ 18us) write_op.cc:260] PREPARE: finished
0317 21:19:24.186963 (+ 120us) write_op.cc:270] Start()
0317 21:19:24.187057 (+ 94us) write_op.cc:276] Timestamp: P: 1773782364186895 usec, L: 0
0317 21:19:24.187076 (+ 19us) op_driver.cc:348] REPLICATION: starting
0317 21:19:24.187655 (+ 579us) log.cc:844] Serialized 728 byte log entry
0317 21:19:26.049586 (+1861931us) op_driver.cc:464] REPLICATION: finished
0317 21:19:26.109289 (+ 59703us) write_op.cc:301] APPLY: starting
0317 21:19:26.109375 (+ 86us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:26.109734 (+ 359us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:26.110005 (+ 271us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:26.110410 (+ 405us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:26.110568 (+ 158us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:26.110629 (+ 61us) write_op.cc:312] APPLY: finished
0317 21:19:26.110838 (+ 209us) log.cc:844] Serialized 29 byte log entry
0317 21:19:26.111017 (+ 179us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:26.111105 (+ 88us) write_op.cc:454] Released schema lock
0317 21:19:26.111143 (+ 38us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":4525,"spinlock_wait_cycles":18900864,"child_traces":[["op",{"apply.queue_time_us":59499,"apply.run_cpu_time_us":2124,"apply.run_wall_time_us":2125,"num_ops":2,"prepare.queue_time_us":129,"prepare.run_cpu_time_us":2335,"prepare.run_wall_time_us":2336,"replication_time_us":1862384,"threads_started":1}]]}
I20260317 21:19:26.128463 8782 raft_consensus.cc:2749] T 788f0ee1f7de47a49c5601c9b1494885 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:26.132501 7467 ts_tablet_manager.cc:1434] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.016s user 0.003s sys 0.004s
I20260317 21:19:26.133477 7467 tablet_bootstrap.cc:492] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.107674 6971 leader_election.cc:304] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:26.122013 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet 4c19bac7772a498f87f48e6095310347 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:24 UTC) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:26.110689 7459 consensus_queue.cc:260] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.138867 8625 raft_consensus.cc:2804] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:26.139240 8625 raft_consensus.cc:493] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:26.139546 8625 raft_consensus.cc:3060] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.133077 8738 raft_consensus.cc:493] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.134358 7036 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:26.135511 7037 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f"
I20260317 21:19:26.140610 7036 raft_consensus.cc:2468] T b76fddff23d04a9a91ea26a366249784 P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:26.141912 8738 raft_consensus.cc:515] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.136767 7034 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e2c52f4843234e8584a66e5e8e065667" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" is_pre_election: true
I20260317 21:19:26.144896 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:26.146530 6971 leader_election.cc:343] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.147480 8738 leader_election.cc:290] T 70472ad137f34237a5a67f74c94aa36b P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.148206 7459 ts_tablet_manager.cc:1434] T dfff4c8478b64436a666a14c133c36d3 P d3cb3b49e3e94537a6e434ea4e037c3f: Time spent starting tablet: real 0.109s user 0.006s sys 0.001s
I20260317 21:19:26.148932 8625 raft_consensus.cc:515] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.149379 6737 catalog_manager.cc:5002] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): tablet 00f34026fd2f4575bb86a9dadb733fe0 (table default.Table_0a9cb20926ea45aaa419518e3f9c57ab [id=c5ec46478903444e8777ecd0a8c4b234]) successfully deleted
I20260317 21:19:26.150571 8780 tablet_replica.cc:333] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f: stopping tablet replica
I20260317 21:19:26.151726 7183 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16f64073188f4d06859e1a701df73242" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a"
I20260317 21:19:26.151921 8625 leader_election.cc:290] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.152282 7183 raft_consensus.cc:3060] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.156769 7467 tablet_bootstrap.cc:654] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.158411 8820 raft_consensus.cc:493] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.158988 8820 raft_consensus.cc:515] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.168748 8820 leader_election.cc:290] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:26.170360 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "eb3a61ed542741d88d89ef6b4c53e9e6" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
W20260317 21:19:26.171659 7261 leader_election.cc:343] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.175212 7183 raft_consensus.cc:2468] T 16f64073188f4d06859e1a701df73242 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 1.
I20260317 21:19:26.175617 7467 tablet_bootstrap.cc:492] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.176057 7467 ts_tablet_manager.cc:1403] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.043s user 0.009s sys 0.001s
I20260317 21:19:26.176327 6971 leader_election.cc:304] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:26.176927 7459 tablet_bootstrap.cc:492] T ae1b2a549af6485ca3faee3f0ad845a2 P d3cb3b49e3e94537a6e434ea4e037c3f: Bootstrap starting.
I20260317 21:19:26.177109 8738 raft_consensus.cc:2804] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:26.177534 8738 raft_consensus.cc:697] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Becoming Leader. State: Replica: d3cb3b49e3e94537a6e434ea4e037c3f, State: Running, Role: LEADER
I20260317 21:19:26.178336 8738 consensus_queue.cc:237] T 16f64073188f4d06859e1a701df73242 P d3cb3b49e3e94537a6e434ea4e037c3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.178735 7467 raft_consensus.cc:359] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.179384 7467 raft_consensus.cc:385] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.179638 7467 raft_consensus.cc:740] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.180235 7467 consensus_queue.cc:260] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.181965 7467 ts_tablet_manager.cc:1434] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260317 21:19:26.182864 7467 tablet_bootstrap.cc:492] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.191583 8827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 21186091f2c9410ea6e585c5ef2c6291. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:26.192360 8827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:26.201026 6750 catalog_manager.cc:5958] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291: Sending DeleteTablet for 3 replicas of tablet b492743dfc4d48398fbae0ee56882aa3
I20260317 21:19:26.199417 7462 tablet_bootstrap.cc:654] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.201493 8818 consensus_queue.cc:1048] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260317 21:19:26.203191 6750 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:50358 (request call id 49) took 2089 ms. Trace:
I20260317 21:19:26.205284 7467 tablet_bootstrap.cc:654] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.205318 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bc2cd63423f844aea29ed3fd839f083e" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.207013 7006 tablet_service.cc:1558] Processing DeleteTablet for tablet b492743dfc4d48398fbae0ee56882aa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:24 UTC) from {username='slave'} at 127.0.0.1:34424
I20260317 21:19:26.203538 6750 rpcz_store.cc:276] 0317 21:19:24.113650 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.113788 (+ 138us) service_pool.cc:225] Handling call
0317 21:19:24.122409 (+ 8621us) spinlock_profiling.cc:243] Waited 7.19 ms on lock 0x7b5400030554. stack: 00007f5fbab7a423 00007f5fbab7a320 00007f5fba6ef574 0000000000509544 00000000005094d9 00007f5fc1bd62c0 00007f5fc1c74e6a 00007f5fc397721c 00007f5fc3a71386 00007f5fc1697024 00007f5fc1696fb2 00007f5fc1696ee1 00007f5fc1696e5c 00007f5fc1696112 00007f5fbf96834c 00007f5fbf967786
0317 21:19:24.123948 (+ 1539us) catalog_manager.cc:2393] Looking up, locking, and authorizing table
0317 21:19:24.150224 (+ 26276us) catalog_manager.cc:2815] Modifying in-memory table state
0317 21:19:24.150374 (+ 150us) catalog_manager.cc:2824] Locking tablets
0317 21:19:24.151226 (+ 852us) catalog_manager.cc:2838] Removing table and tablets from system table
0317 21:19:26.200487 (+2049261us) catalog_manager.cc:2856] Removing table from by-name map
0317 21:19:26.200761 (+ 274us) catalog_manager.cc:2872] Committing in-memory state
0317 21:19:26.200848 (+ 87us) catalog_manager.cc:2876] Aborting table tasks
0317 21:19:26.203128 (+ 2280us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:24.195400 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:24.195873 (+ 473us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:24.195900 (+ 27us) write_op.cc:435] Acquired schema lock
0317 21:19:24.195916 (+ 16us) tablet.cc:661] Decoding operations
0317 21:19:24.196277 (+ 361us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:24.196319 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:24.196351 (+ 32us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:24.196695 (+ 344us) tablet.cc:700] Row locks acquired
0317 21:19:24.196714 (+ 19us) write_op.cc:260] PREPARE: finished
0317 21:19:24.196844 (+ 130us) write_op.cc:270] Start()
0317 21:19:24.196935 (+ 91us) write_op.cc:276] Timestamp: P: 1773782364196776 usec, L: 0
0317 21:19:24.196965 (+ 30us) op_driver.cc:348] REPLICATION: starting
0317 21:19:24.197491 (+ 526us) log.cc:844] Serialized 728 byte log entry
0317 21:19:26.163638 (+1966147us) op_driver.cc:464] REPLICATION: finished
0317 21:19:26.166280 (+ 2642us) write_op.cc:301] APPLY: starting
0317 21:19:26.166350 (+ 70us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:26.166753 (+ 403us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:26.178916 (+ 12163us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:26.179190 (+ 274us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:26.179325 (+ 135us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0317 21:19:26.179369 (+ 44us) write_op.cc:312] APPLY: finished
0317 21:19:26.179592 (+ 223us) log.cc:844] Serialized 29 byte log entry
0317 21:19:26.179756 (+ 164us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:26.179838 (+ 82us) write_op.cc:454] Released schema lock
0317 21:19:26.179869 (+ 31us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":16546688,"child_traces":[["op",{"apply.queue_time_us":2454,"num_ops":2,"prepare.queue_time_us":116,"prepare.run_cpu_time_us":2328,"prepare.run_wall_time_us":2329,"replication_time_us":1966543}]]}
I20260317 21:19:26.209467 7151 tablet_service.cc:1558] Processing DeleteTablet for tablet b492743dfc4d48398fbae0ee56882aa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:24 UTC) from {username='slave'} at 127.0.0.1:48088
I20260317 21:19:26.210772 8620 raft_consensus.cc:493] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.211288 8620 raft_consensus.cc:515] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.214841 6747 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50358 (request call id 48) took 2102 ms. Trace:
I20260317 21:19:26.215149 8620 leader_election.cc:290] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.215186 6747 rpcz_store.cc:276] 0317 21:19:24.112185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.112327 (+ 142us) service_pool.cc:225] Handling call
0317 21:19:24.141126 (+ 28799us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x7b5400030554. stack: 00007f5fbab7a423 00007f5fbab7a320 00007f5fba6ef574 0000000000509544 00000000005094d9 00007f5fc1bd62c0 00007f5fc1c74e6a 00007f5fc397721c 00007f5fc3a71076 00007f5fc1692524 00007f5fc16924b2 00007f5fc16923e1 00007f5fc169235c 00007f5fc1691612 00007f5fbf96834c 00007f5fbf967786
0317 21:19:24.143825 (+ 2699us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:24.196988 (+ 53163us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:26.213806 (+2016818us) catalog_manager.cc:2207] Wrote table and tablets to system table
0317 21:19:26.214170 (+ 364us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0317 21:19:26.214633 (+ 463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:24.198468 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:24.198939 (+ 471us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:24.198963 (+ 24us) write_op.cc:435] Acquired schema lock
0317 21:19:24.198976 (+ 13us) tablet.cc:661] Decoding operations
0317 21:19:24.199293 (+ 317us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:24.199330 (+ 37us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:24.199353 (+ 23us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:24.199576 (+ 223us) tablet.cc:700] Row locks acquired
0317 21:19:24.199588 (+ 12us) write_op.cc:260] PREPARE: finished
0317 21:19:24.199710 (+ 122us) write_op.cc:270] Start()
0317 21:19:24.199803 (+ 93us) write_op.cc:276] Timestamp: P: 1773782364199641 usec, L: 0
0317 21:19:24.199821 (+ 18us) op_driver.cc:348] REPLICATION: starting
0317 21:19:24.200403 (+ 582us) log.cc:844] Serialized 433 byte log entry
0317 21:19:26.163929 (+1963526us) op_driver.cc:464] REPLICATION: finished
0317 21:19:26.177729 (+ 13800us) write_op.cc:301] APPLY: starting
0317 21:19:26.177803 (+ 74us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:26.177912 (+ 109us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:26.178014 (+ 102us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:26.178475 (+ 461us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:26.212701 (+ 34226us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:26.212900 (+ 199us) write_op.cc:312] APPLY: finished
0317 21:19:26.213247 (+ 347us) log.cc:844] Serialized 29 byte log entry
0317 21:19:26.213388 (+ 141us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:26.213462 (+ 74us) write_op.cc:454] Released schema lock
0317 21:19:26.213509 (+ 47us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":34722688,"child_traces":[["op",{"apply.queue_time_us":13644,"num_ops":2,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":2156,"prepare.run_wall_time_us":2156,"replication_time_us":1964006,"thread_start_us":12231,"threads_started":1}]]}
W20260317 21:19:26.219007 6973 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.67:32877, user_credentials={real_user=slave}} blocked reactor thread for 82150.7us
I20260317 21:19:26.220299 7120 leader_election.cc:304] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f; no voters:
I20260317 21:19:26.220597 6633 master-stress-test.cc:579] Killed test thread 2/10
I20260317 21:19:26.221040 6633 master-stress-test.cc:577] Killing test thread 3/10
I20260317 21:19:26.221468 8706 raft_consensus.cc:2804] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:26.221848 8706 raft_consensus.cc:493] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:26.222198 8706 raft_consensus.cc:3060] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.222671 8780 raft_consensus.cc:2243] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:26.223799 8780 raft_consensus.cc:2272] T 7fc38947219047b5abdf71af5554a987 P d3cb3b49e3e94537a6e434ea4e037c3f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:26.232389 8706 raft_consensus.cc:515] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.239701 8824 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:26.240437 8824 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:26.240701 7183 raft_consensus.cc:1275] T a4d64c48641b484ab443380808544d2f P c0ef92fef382401a9564c17e0c223f6a [term 2 FOLLOWER]: Refusing update from remote peer 2f539d1a4a584b8ca83928be49369bf4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260317 21:19:26.250476 8706 leader_election.cc:290] T b76fddff23d04a9a91ea26a366249784 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.256012 6745 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50358 (request call id 51) took 2138 ms. Trace:
I20260317 21:19:26.259578 7459 tablet_bootstrap.cc:654] T ae1b2a549af6485ca3faee3f0ad845a2 P d3cb3b49e3e94537a6e434ea4e037c3f: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.256542 6745 rpcz_store.cc:276] 0317 21:19:24.117032 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.117206 (+ 174us) service_pool.cc:225] Handling call
0317 21:19:24.133973 (+ 16767us) spinlock_profiling.cc:243] Waited 9.81 ms on lock 0x7b5400030554. stack: 00007f5fbab7a423 00007f5fbab7a320 00007f5fba6ef574 0000000000509544 00000000005094d9 00007f5fc1bd62c0 00007f5fc1c74e6a 00007f5fc397721c 00007f5fc3a71076 00007f5fc1692524 00007f5fc16924b2 00007f5fc16923e1 00007f5fc169235c 00007f5fc1691612 00007f5fbf96834c 00007f5fbf967786
0317 21:19:24.204835 (+ 70862us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:24.234311 (+ 29476us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:26.254771 (+2020460us) catalog_manager.cc:2207] Wrote table and tablets to system table
0317 21:19:26.255297 (+ 526us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0317 21:19:26.255804 (+ 507us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:24.238077 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:24.238556 (+ 479us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:24.238582 (+ 26us) write_op.cc:435] Acquired schema lock
0317 21:19:24.238599 (+ 17us) tablet.cc:661] Decoding operations
0317 21:19:24.238941 (+ 342us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:24.238982 (+ 41us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:24.239026 (+ 44us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:24.239318 (+ 292us) tablet.cc:700] Row locks acquired
0317 21:19:24.239343 (+ 25us) write_op.cc:260] PREPARE: finished
0317 21:19:24.239525 (+ 182us) write_op.cc:270] Start()
0317 21:19:24.239653 (+ 128us) write_op.cc:276] Timestamp: P: 1773782364239425 usec, L: 0
0317 21:19:24.239680 (+ 27us) op_driver.cc:348] REPLICATION: starting
0317 21:19:24.240474 (+ 794us) log.cc:844] Serialized 433 byte log entry
0317 21:19:26.191668 (+1951194us) op_driver.cc:464] REPLICATION: finished
0317 21:19:26.210340 (+ 18672us) write_op.cc:301] APPLY: starting
0317 21:19:26.210416 (+ 76us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:26.210635 (+ 219us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:26.210743 (+ 108us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:26.211275 (+ 532us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:26.211412 (+ 137us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:26.211457 (+ 45us) write_op.cc:312] APPLY: finished
0317 21:19:26.211655 (+ 198us) log.cc:844] Serialized 29 byte log entry
0317 21:19:26.211814 (+ 159us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:26.211875 (+ 61us) write_op.cc:454] Released schema lock
0317 21:19:26.211910 (+ 35us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":22554368,"child_traces":[["op",{"apply.queue_time_us":18503,"num_ops":2,"prepare.queue_time_us":2044,"prepare.run_cpu_time_us":2541,"prepare.run_wall_time_us":4405,"replication_time_us":1951833,"thread_start_us":44305,"threads_started":1}]]}
I20260317 21:19:26.264267 6633 master-stress-test.cc:579] Killed test thread 3/10
I20260317 21:19:26.264561 6633 master-stress-test.cc:577] Killing test thread 4/10
I20260317 21:19:26.265610 6746 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:50358 (request call id 50) took 2150 ms. Trace:
I20260317 21:19:26.265981 7300 tablet_service.cc:1558] Processing DeleteTablet for tablet b492743dfc4d48398fbae0ee56882aa3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-03-17 21:19:24 UTC) from {username='slave'} at 127.0.0.1:33804
I20260317 21:19:26.266844 8820 consensus_queue.cc:1048] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:26.266095 6746 rpcz_store.cc:276] 0317 21:19:24.114662 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.115067 (+ 405us) service_pool.cc:225] Handling call
0317 21:19:24.145174 (+ 30107us) spinlock_profiling.cc:243] Waited 11.5 ms on lock 0x7b5400030554. stack: 00007f5fbab7a423 00007f5fbab7a320 00007f5fba6ef574 0000000000509544 00000000005094d9 00007f5fc1bd62c0 00007f5fc1c74e6a 00007f5fc397721c 00007f5fc3a71076 00007f5fc1692524 00007f5fc16924b2 00007f5fc16923e1 00007f5fc169235c 00007f5fc1691612 00007f5fbf96834c 00007f5fbf967786
0317 21:19:24.148428 (+ 3254us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:19:24.199290 (+ 50862us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:19:26.263807 (+2064517us) catalog_manager.cc:2207] Wrote table and tablets to system table
0317 21:19:26.264275 (+ 468us) catalog_manager.cc:2241] Inserted table and tablets into CatalogManager maps
0317 21:19:26.264918 (+ 643us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0317 21:19:24.233448 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0317 21:19:24.233931 (+ 483us) write_op.cc:432] Acquiring schema lock in shared mode
0317 21:19:24.233961 (+ 30us) write_op.cc:435] Acquired schema lock
0317 21:19:24.233978 (+ 17us) tablet.cc:661] Decoding operations
0317 21:19:24.234314 (+ 336us) write_op.cc:620] Acquiring the partition lock for write op
0317 21:19:24.234356 (+ 42us) write_op.cc:641] Partition lock acquired for write op
0317 21:19:24.234382 (+ 26us) tablet.cc:684] Acquiring locks for 2 operations
0317 21:19:24.234601 (+ 219us) tablet.cc:700] Row locks acquired
0317 21:19:24.234617 (+ 16us) write_op.cc:260] PREPARE: finished
0317 21:19:24.234764 (+ 147us) write_op.cc:270] Start()
0317 21:19:24.234851 (+ 87us) write_op.cc:276] Timestamp: P: 1773782364234690 usec, L: 0
0317 21:19:24.234869 (+ 18us) op_driver.cc:348] REPLICATION: starting
0317 21:19:24.235405 (+ 536us) log.cc:844] Serialized 433 byte log entry
0317 21:19:26.176567 (+1941162us) op_driver.cc:464] REPLICATION: finished
0317 21:19:26.198507 (+ 21940us) write_op.cc:301] APPLY: starting
0317 21:19:26.198704 (+ 197us) tablet.cc:1366] starting BulkCheckPresence
0317 21:19:26.198816 (+ 112us) tablet.cc:1369] finished BulkCheckPresence
0317 21:19:26.198950 (+ 134us) tablet.cc:1371] starting ApplyRowOperation cycle
0317 21:19:26.199524 (+ 574us) tablet.cc:1382] finished ApplyRowOperation cycle
0317 21:19:26.199688 (+ 164us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0317 21:19:26.199729 (+ 41us) write_op.cc:312] APPLY: finished
0317 21:19:26.199948 (+ 219us) log.cc:844] Serialized 29 byte log entry
0317 21:19:26.200099 (+ 151us) write_op.cc:489] Releasing partition, row and schema locks
0317 21:19:26.263356 (+ 63257us) write_op.cc:454] Released schema lock
0317 21:19:26.263426 (+ 70us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":26381184,"child_traces":[["op",{"apply.queue_time_us":21561,"num_ops":2,"prepare.queue_time_us":263,"prepare.run_cpu_time_us":2190,"prepare.run_wall_time_us":2189,"replication_time_us":1941531,"thread_start_us":14435,"threads_started":1}]]}
I20260317 21:19:26.273694 8792 ts_tablet_manager.cc:1929] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:26.273731 6633 master-stress-test.cc:579] Killed test thread 4/10
I20260317 21:19:26.274083 6633 master-stress-test.cc:577] Killing test thread 5/10
I20260317 21:19:26.274233 8792 log.cc:1199] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/8a44a297079347bab6f8be6abe332bfa
I20260317 21:19:26.274693 6633 master-stress-test.cc:579] Killed test thread 5/10
I20260317 21:19:26.274933 6633 master-stress-test.cc:577] Killing test thread 6/10
I20260317 21:19:26.274941 8792 ts_tablet_manager.cc:1950] T 8a44a297079347bab6f8be6abe332bfa P 2f539d1a4a584b8ca83928be49369bf4: Deleting consensus metadata
I20260317 21:19:26.275255 6633 master-stress-test.cc:579] Killed test thread 6/10
I20260317 21:19:26.275467 6633 master-stress-test.cc:577] Killing test thread 7/10
I20260317 21:19:26.275772 6633 master-stress-test.cc:579] Killed test thread 7/10
I20260317 21:19:26.276010 6633 master-stress-test.cc:577] Killing test thread 8/10
I20260317 21:19:26.276306 6633 master-stress-test.cc:579] Killed test thread 8/10
I20260317 21:19:26.277441 6633 master-stress-test.cc:577] Killing test thread 9/10
I20260317 21:19:26.277832 6633 master-stress-test.cc:579] Killed test thread 9/10
I20260317 21:19:26.278064 6633 master-stress-test.cc:577] Killing test thread 10/10
I20260317 21:19:26.278458 6633 master-stress-test.cc:579] Killed test thread 10/10
I20260317 21:19:26.278903 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 6919
I20260317 21:19:26.280234 7320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.281026 7320 raft_consensus.cc:2468] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3cb3b49e3e94537a6e434ea4e037c3f in term 0.
I20260317 21:19:26.284557 8827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:26.285308 8827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:26.303161 8792 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:33804 (request call id 3) took 1181 ms. Trace:
I20260317 21:19:26.303684 8792 rpcz_store.cc:276] 0317 21:19:25.121801 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:25.121929 (+ 128us) service_pool.cc:225] Handling call
0317 21:19:25.899711 (+777782us) ts_tablet_manager.cc:1151] Deleting tablet 8a44a297079347bab6f8be6abe332bfa
0317 21:19:25.899759 (+ 48us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:26.051409 (+151650us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.272554 (+221145us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.303050 (+ 30496us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":777276}
I20260317 21:19:26.309159 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:26.309909 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:26.310766 8810 sys_catalog.cc:455] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 10 leader_uuid: "21186091f2c9410ea6e585c5ef2c6291" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } } }
I20260317 21:19:26.311388 8810 sys_catalog.cc:458] T 00000000000000000000000000000000 P 21186091f2c9410ea6e585c5ef2c6291 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:26.351138 8786 ts_tablet_manager.cc:1929] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:26.362603 8786 log.cc:1199] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/00f34026fd2f4575bb86a9dadb733fe0
I20260317 21:19:26.363610 8786 ts_tablet_manager.cc:1950] T 00f34026fd2f4575bb86a9dadb733fe0 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:26.370564 7467 tablet_bootstrap.cc:492] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.371008 7467 ts_tablet_manager.cc:1403] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.188s user 0.021s sys 0.008s
I20260317 21:19:26.374157 8786 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48088 (request call id 1) took 1439 ms. Trace:
I20260317 21:19:26.396950 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:26.397984 7329 raft_consensus.cc:2393] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:26.399046 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b76fddff23d04a9a91ea26a366249784" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.400008 7322 raft_consensus.cc:2393] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate c0ef92fef382401a9564c17e0c223f6a in current term 1: Already voted for candidate 2f539d1a4a584b8ca83928be49369bf4 in this term.
I20260317 21:19:26.408408 7462 tablet_bootstrap.cc:492] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:26.409174 7462 ts_tablet_manager.cc:1403] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.405s user 0.010s sys 0.002s
I20260317 21:19:26.393838 7467 raft_consensus.cc:359] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.410468 7467 raft_consensus.cc:385] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.410825 7467 raft_consensus.cc:740] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.410375 8786 rpcz_store.cc:276] 0317 21:19:24.934340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.935406 (+ 1066us) service_pool.cc:225] Handling call
0317 21:19:26.033168 (+1097762us) ts_tablet_manager.cc:1151] Deleting tablet 00f34026fd2f4575bb86a9dadb733fe0
0317 21:19:26.060716 (+ 27548us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:26.309332 (+248616us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.350864 (+ 41532us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.374083 (+ 23219us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1083467}
I20260317 21:19:26.411805 6734 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 00f34026fd2f4575bb86a9dadb733fe0 (table default.Table_0a9cb20926ea45aaa419518e3f9c57ab [id=c5ec46478903444e8777ecd0a8c4b234]) successfully deleted
I20260317 21:19:26.411592 7467 consensus_queue.cc:260] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.413800 7467 ts_tablet_manager.cc:1434] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.043s user 0.005s sys 0.002s
I20260317 21:19:26.423858 8786 tablet_replica.cc:333] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:26.424329 7467 tablet_bootstrap.cc:492] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.425006 8786 raft_consensus.cc:2243] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:19:26.425962 8786 raft_consensus.cc:2272] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Raft consensus is shut down!
W20260317 21:19:26.432221 7261 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.6.122.66:45979, user_credentials={real_user=slave}} blocked reactor thread for 46103.8us
I20260317 21:19:26.435418 8782 raft_consensus.cc:493] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.441284 7119 leader_election.cc:304] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c0ef92fef382401a9564c17e0c223f6a; no voters: 2f539d1a4a584b8ca83928be49369bf4, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:26.436148 8782 raft_consensus.cc:515] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.451244 8782 leader_election.cc:290] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:26.453999 8674 raft_consensus.cc:2749] T bc2cd63423f844aea29ed3fd839f083e P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:26.454144 7119 leader_election.cc:343] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): Not found: Tablet not found: 8a44a297079347bab6f8be6abe332bfa
I20260317 21:19:26.458047 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f5027b892229491eb5a829cacd29a700" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:26.465328 8786 ts_tablet_manager.cc:1916] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:26.474512 7467 tablet_bootstrap.cc:654] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.446215 7462 raft_consensus.cc:359] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.477607 7462 raft_consensus.cc:385] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.478122 7462 raft_consensus.cc:740] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
W20260317 21:19:26.477011 7261 leader_election.cc:343] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.480760 7462 consensus_queue.cc:260] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.518972 7467 tablet_bootstrap.cc:492] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.519529 7467 ts_tablet_manager.cc:1403] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.000s
I20260317 21:19:26.522089 7467 raft_consensus.cc:359] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.522826 7467 raft_consensus.cc:385] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.523173 7467 raft_consensus.cc:740] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.523936 7467 consensus_queue.cc:260] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.537813 7467 ts_tablet_manager.cc:1434] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.018s user 0.005s sys 0.000s
I20260317 21:19:26.541102 8850 raft_consensus.cc:493] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.541664 8850 raft_consensus.cc:515] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.543603 8850 leader_election.cc:290] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:26.556542 7264 leader_election.cc:304] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
W20260317 21:19:26.557997 7264 leader_election.cc:343] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.558097 8781 raft_consensus.cc:2749] T b76fddff23d04a9a91ea26a366249784 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260317 21:19:26.558475 7264 leader_election.cc:304] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:26.559410 8781 raft_consensus.cc:2749] T e2c52f4843234e8584a66e5e8e065667 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:26.564538 8824 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:26.565083 8824 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:26.566989 8824 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:26.569072 6685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 11 candidate_status { last_received { term: 10 index: 430 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:26.570653 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 11 candidate_status { last_received { term: 10 index: 430 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:26.572839 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
I20260317 21:19:26.574179 8824 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260317 21:19:26.575757 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "17610bdee98f49c4b1d8b5481867c6e8" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:26.575960 7467 tablet_bootstrap.cc:492] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.593787 7467 tablet_bootstrap.cc:654] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.595394 7462 ts_tablet_manager.cc:1434] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.186s user 0.012s sys 0.000s
I20260317 21:19:26.610658 6736 catalog_manager.cc:5002] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): tablet 8a44a297079347bab6f8be6abe332bfa (table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]) successfully deleted
W20260317 21:19:26.611989 6736 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet 4c19bac7772a498f87f48e6095310347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c19bac7772a498f87f48e6095310347 already in progress: creating tablet
W20260317 21:19:26.613209 6736 catalog_manager.cc:4982] TS 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877): delete failed for tablet b492743dfc4d48398fbae0ee56882aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b492743dfc4d48398fbae0ee56882aa3 already in progress: creating tablet
I20260317 21:19:26.615780 8620 raft_consensus.cc:493] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.616251 8620 raft_consensus.cc:515] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.619131 8620 leader_election.cc:290] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.621577 7467 tablet_bootstrap.cc:492] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.621994 7467 ts_tablet_manager.cc:1403] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.046s user 0.008s sys 0.000s
I20260317 21:19:26.630966 7467 raft_consensus.cc:359] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.631642 7467 raft_consensus.cc:385] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.631917 7467 raft_consensus.cc:740] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.632570 7467 consensus_queue.cc:260] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.635030 7467 ts_tablet_manager.cc:1434] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
I20260317 21:19:26.636116 7467 tablet_bootstrap.cc:492] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.643963 7467 tablet_bootstrap.cc:654] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.635030 8786 ts_tablet_manager.cc:1929] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:19:26.656859 8786 log.cc:1199] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/fc8567ae10464694a1128ba2eb1d8ee1
I20260317 21:19:26.657862 8786 ts_tablet_manager.cc:1950] T fc8567ae10464694a1128ba2eb1d8ee1 P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:26.622516 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfff4c8478b64436a666a14c133c36d3" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.665066 7329 raft_consensus.cc:2468] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:26.630152 8818 raft_consensus.cc:493] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.666031 8818 raft_consensus.cc:515] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.668310 8818 leader_election.cc:290] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:26.669713 7119 leader_election.cc:304] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:26.670475 8620 raft_consensus.cc:2804] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:26.670819 8620 raft_consensus.cc:493] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:26.671146 8620 raft_consensus.cc:3060] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:26.674556 7263 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.6.122.125:32779, user_credentials={real_user=slave}} blocked reactor thread for 51168.3us
W20260317 21:19:26.675853 8853 log.cc:927] Time spent T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4: Append to log took a long time: real 0.061s user 0.001s sys 0.000s
I20260317 21:19:26.660348 8786 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48088 (request call id 2) took 1725 ms. Trace:
I20260317 21:19:26.687217 7462 tablet_bootstrap.cc:492] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:26.687486 7177 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "6ab5f3dc7836425d917bd09f1e6bfe90" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:26.702572 8620 raft_consensus.cc:515] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.705881 8620 leader_election.cc:290] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.708330 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "dfff4c8478b64436a666a14c133c36d3" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:26.708914 7329 raft_consensus.cc:3060] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.679093 8786 rpcz_store.cc:276] 0317 21:19:24.934629 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.936057 (+ 1428us) service_pool.cc:225] Handling call
0317 21:19:26.414965 (+1478908us) ts_tablet_manager.cc:1151] Deleting tablet fc8567ae10464694a1128ba2eb1d8ee1
0317 21:19:26.423718 (+ 8753us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:26.549687 (+125969us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.634779 (+ 85092us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.660281 (+ 25502us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1465052}
I20260317 21:19:26.714150 8786 tablet_replica.cc:333] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: stopping tablet replica
I20260317 21:19:26.714411 6734 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet fc8567ae10464694a1128ba2eb1d8ee1 (table default.Table_1885223af9524a43b0d34db30e68e6f1 [id=1293ecb1a2c24e7e987c4c0ed2524a94]) successfully deleted
I20260317 21:19:26.715032 8786 raft_consensus.cc:2243] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:19:26.715508 8786 raft_consensus.cc:2272] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:19:26.717417 8786 ts_tablet_manager.cc:1916] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:19:26.717808 7329 raft_consensus.cc:2468] T dfff4c8478b64436a666a14c133c36d3 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:26.718418 8620 raft_consensus.cc:493] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.719007 8620 raft_consensus.cc:515] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.721182 8620 leader_election.cc:290] T d4afbb1ab93a44d9ab5795b72444c690 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.723032 7119 leader_election.cc:304] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:26.723956 8620 raft_consensus.cc:2804] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:26.724551 8620 raft_consensus.cc:697] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:26.725657 8620 consensus_queue.cc:237] T dfff4c8478b64436a666a14c133c36d3 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.730545 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d4afbb1ab93a44d9ab5795b72444c690" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
W20260317 21:19:26.740620 7261 leader_election.cc:343] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:19:26.742151 7261 leader_election.cc:343] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:26.753631 8827 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:26.754168 8827 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:26.756122 8827 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
W20260317 21:19:26.757248 6661 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet e06a1488c60f4d1387417ce30579fbf4 (table default.Table_54698fcc6a7a46deaba83055d4a1c78d [id=3e906ea0e8af47e383d0403c155be644], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e06a1488c60f4d1387417ce30579fbf4 consensus state has no leader
I20260317 21:19:26.758571 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 11 candidate_status { last_received { term: 10 index: 430 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:26.759773 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 11 candidate_status { last_received { term: 10 index: 430 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:26.761319 6662 leader_election.cc:304] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 31726fd7f40442e49b2fe0f880793206; no voters: 21186091f2c9410ea6e585c5ef2c6291, 47e1c983d0224428a9a7b147a0e5e4ef
I20260317 21:19:26.762941 8827 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260317 21:19:26.790665 7151 tablet_service.cc:1511] Processing CreateTablet for tablet 343047b78fb345729c6b66454e2cab40 (DEFAULT_TABLE table=default.Table_7864164e6f474a35a89094d36d4254ec [id=f2daa1d797d547ff9c87f31e4560581d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.791926 7151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 343047b78fb345729c6b66454e2cab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.795496 7162 tablet_service.cc:1511] Processing CreateTablet for tablet d85227a2e355424d92e605dd53d9fd14 (DEFAULT_TABLE table=default.Table_9c887813658144a5ab397c605d8954c0 [id=def26fdbf280448380ad71d8c52b9713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.795742 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 343047b78fb345729c6b66454e2cab40 (DEFAULT_TABLE table=default.Table_7864164e6f474a35a89094d36d4254ec [id=f2daa1d797d547ff9c87f31e4560581d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.796876 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85227a2e355424d92e605dd53d9fd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.797097 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 343047b78fb345729c6b66454e2cab40. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.805333 7467 tablet_bootstrap.cc:492] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.805795 7467 ts_tablet_manager.cc:1403] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.170s user 0.014s sys 0.006s
I20260317 21:19:26.811098 8620 raft_consensus.cc:493] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.811609 8620 raft_consensus.cc:515] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.813750 8620 leader_election.cc:290] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.815536 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.816185 7329 raft_consensus.cc:2468] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:26.808173 7467 raft_consensus.cc:359] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.816737 7467 raft_consensus.cc:385] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.817052 7467 raft_consensus.cc:740] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.817230 7119 leader_election.cc:304] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:26.817796 7467 consensus_queue.cc:260] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.818636 8620 raft_consensus.cc:2804] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:26.819023 8620 raft_consensus.cc:493] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:26.819350 8620 raft_consensus.cc:3060] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.820289 7467 ts_tablet_manager.cc:1434] T 4af622a9190148ee80b2d5d3d7a92a20 P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.014s user 0.006s sys 0.000s
I20260317 21:19:26.837253 7462 tablet_bootstrap.cc:654] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.841092 7306 tablet_service.cc:1511] Processing CreateTablet for tablet d85227a2e355424d92e605dd53d9fd14 (DEFAULT_TABLE table=default.Table_9c887813658144a5ab397c605d8954c0 [id=def26fdbf280448380ad71d8c52b9713]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.842324 7306 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d85227a2e355424d92e605dd53d9fd14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.857266 7467 tablet_bootstrap.cc:492] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.858547 8786 ts_tablet_manager.cc:1929] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:19:26.858959 8786 log.cc:1199] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/8a44a297079347bab6f8be6abe332bfa
I20260317 21:19:26.874366 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 1833d3ee7bc948ffb34418027b72b0ea (DEFAULT_TABLE table=default.Table_a4ca7b6127064c19b73aee408bc49282 [id=f1ff0d130d854a349d415b1d7d91a836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.881774 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 1833d3ee7bc948ffb34418027b72b0ea (DEFAULT_TABLE table=default.Table_a4ca7b6127064c19b73aee408bc49282 [id=f1ff0d130d854a349d415b1d7d91a836]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.859645 8786 ts_tablet_manager.cc:1950] T 8a44a297079347bab6f8be6abe332bfa P c0ef92fef382401a9564c17e0c223f6a: Deleting consensus metadata
I20260317 21:19:26.884788 7156 tablet_service.cc:1511] Processing CreateTablet for tablet 5536ae492ac5492b8349cbadd4a156c9 (DEFAULT_TABLE table=default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 [id=b1d87190d48c47288ea093aa65a528a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.876398 8620 raft_consensus.cc:515] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.893488 7156 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5536ae492ac5492b8349cbadd4a156c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.897491 7303 tablet_service.cc:1511] Processing CreateTablet for tablet 5536ae492ac5492b8349cbadd4a156c9 (DEFAULT_TABLE table=default.Table_ebd4ccb1fd914f3abdc9fdaa12e7dac7 [id=b1d87190d48c47288ea093aa65a528a0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.883965 7467 tablet_bootstrap.cc:654] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:26.906948 8706 raft_consensus.cc:493] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:26.907534 8706 raft_consensus.cc:515] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.909963 8706 leader_election.cc:290] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.910943 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4737071c2aec4449a05bead6af7ad32b" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:26.911554 7329 raft_consensus.cc:2468] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:26.912122 8786 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48088 (request call id 3) took 1977 ms. Trace:
I20260317 21:19:26.912544 7119 leader_election.cc:304] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:26.912590 7151 tablet_service.cc:1511] Processing CreateTablet for tablet 516540c0439d4b75a3ac1689b8964bf7 (DEFAULT_TABLE table=default.Table_6254d435474446bca35e4c78478b186f [id=a796a06b6825482b969b97c9da0ab79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.912922 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 516540c0439d4b75a3ac1689b8964bf7 (DEFAULT_TABLE table=default.Table_6254d435474446bca35e4c78478b186f [id=a796a06b6825482b969b97c9da0ab79d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.912469 8786 rpcz_store.cc:276] 0317 21:19:24.934903 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:19:24.936666 (+ 1763us) service_pool.cc:225] Handling call
0317 21:19:26.713986 (+1777320us) ts_tablet_manager.cc:1151] Deleting tablet 8a44a297079347bab6f8be6abe332bfa
0317 21:19:26.714037 (+ 51us) ts_tablet_manager.cc:1071] Acquired tablet manager lock
0317 21:19:26.793826 (+ 79789us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.858290 (+ 64464us) tablet_metadata.cc:641] Metadata flushed
0317 21:19:26.912058 (+ 53768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763986}
I20260317 21:19:26.914284 8706 raft_consensus.cc:2804] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:26.914501 6734 catalog_manager.cc:5002] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): tablet 8a44a297079347bab6f8be6abe332bfa (table default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]) successfully deleted
I20260317 21:19:26.914815 8706 raft_consensus.cc:493] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:26.915446 8706 raft_consensus.cc:3060] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:26.916429 6734 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet 4c19bac7772a498f87f48e6095310347 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c19bac7772a498f87f48e6095310347 already in progress: creating tablet
W20260317 21:19:26.917879 6734 catalog_manager.cc:4982] TS c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): delete failed for tablet b492743dfc4d48398fbae0ee56882aa3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b492743dfc4d48398fbae0ee56882aa3 already in progress: creating tablet
I20260317 21:19:26.920689 7150 tablet_service.cc:1511] Processing CreateTablet for tablet 21b1c7a203ef4c22b3a84631cadfa62a (DEFAULT_TABLE table=default.Table_7fee5e45b4aa4a93900162f41e3a3d60 [id=a73a2c0e2c814324ba832290ee403c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.922967 7309 tablet_service.cc:1511] Processing CreateTablet for tablet 21b1c7a203ef4c22b3a84631cadfa62a (DEFAULT_TABLE table=default.Table_7fee5e45b4aa4a93900162f41e3a3d60 [id=a73a2c0e2c814324ba832290ee403c4a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.922828 8818 consensus_queue.cc:1048] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:19:26.927721 7161 tablet_service.cc:1511] Processing CreateTablet for tablet 6deece45839849088dad3d873cc925b2 (DEFAULT_TABLE table=default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.933647 7467 tablet_bootstrap.cc:492] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:26.933475 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 6deece45839849088dad3d873cc925b2 (DEFAULT_TABLE table=default.Table_154f9dc4f8214c7b832788cb604a6f70 [id=81dc7f934d2346cd8ad83e5d718c5d7d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.934322 7467 ts_tablet_manager.cc:1403] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.077s user 0.012s sys 0.000s
I20260317 21:19:26.936856 7164 tablet_service.cc:1511] Processing CreateTablet for tablet 8a2df0b2a73f4ed4b349efbb4d424afa (DEFAULT_TABLE table=default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.935627 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1833d3ee7bc948ffb34418027b72b0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.939095 7161 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6deece45839849088dad3d873cc925b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.942447 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1833d3ee7bc948ffb34418027b72b0ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.945066 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2df0b2a73f4ed4b349efbb4d424afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.945815 7303 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5536ae492ac5492b8349cbadd4a156c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.936903 7467 raft_consensus.cc:359] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.949486 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516540c0439d4b75a3ac1689b8964bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.949487 7467 raft_consensus.cc:385] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:26.949944 7467 raft_consensus.cc:740] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:26.952101 7309 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b1c7a203ef4c22b3a84631cadfa62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.952915 7150 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 21b1c7a203ef4c22b3a84631cadfa62a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.954362 7152 tablet_service.cc:1511] Processing CreateTablet for tablet 7d19b386f49a420f8561dd9c2f415aa4 (DEFAULT_TABLE table=default.Table_4827867befa741f69f6ab15938cfe436 [id=5b18e88ee5b64c1fb5f7e361c866f44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.954631 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 8a2df0b2a73f4ed4b349efbb4d424afa (DEFAULT_TABLE table=default.Table_cb3946746d5d4e008932e6f7f476a59a [id=7446e8fdc04b4b9eb0d84f3a8ddfdd7a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.955631 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8a2df0b2a73f4ed4b349efbb4d424afa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.956142 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 7d19b386f49a420f8561dd9c2f415aa4 (DEFAULT_TABLE table=default.Table_4827867befa741f69f6ab15938cfe436 [id=5b18e88ee5b64c1fb5f7e361c866f44a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:26.957231 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d19b386f49a420f8561dd9c2f415aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.957567 7152 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d19b386f49a420f8561dd9c2f415aa4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.958164 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6deece45839849088dad3d873cc925b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.955617 7467 consensus_queue.cc:260] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:26.962919 8620 leader_election.cc:290] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:26.963089 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70472ad137f34237a5a67f74c94aa36b" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:26.963614 7329 raft_consensus.cc:3060] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:26.967237 7467 ts_tablet_manager.cc:1434] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.033s user 0.006s sys 0.000s
I20260317 21:19:26.978487 7467 tablet_bootstrap.cc:492] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:26.980772 7151 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 516540c0439d4b75a3ac1689b8964bf7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:26.997990 7467 tablet_bootstrap.cc:654] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.006775 7301 tablet_service.cc:1511] Processing CreateTablet for tablet 3de1582e47f948cfa6c3c0c997ea573f (DEFAULT_TABLE table=default.Table_9967a6acfcf84b2c9759c612f25702e6 [id=790dbba0c4ed469a8d5e208209cb5f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.008206 7301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de1582e47f948cfa6c3c0c997ea573f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.021602 8818 raft_consensus.cc:493] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.022109 8818 raft_consensus.cc:515] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.024466 8818 leader_election.cc:290] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:26.999692 8706 raft_consensus.cc:515] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.003377 7159 tablet_service.cc:1511] Processing CreateTablet for tablet 3de1582e47f948cfa6c3c0c997ea573f (DEFAULT_TABLE table=default.Table_9967a6acfcf84b2c9759c612f25702e6 [id=790dbba0c4ed469a8d5e208209cb5f39]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.026072 7462 tablet_bootstrap.cc:492] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:27.026036 7178 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "646a115e133b43eebc1674a862ab7838" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:27.026538 7462 ts_tablet_manager.cc:1403] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.340s user 0.011s sys 0.000s
W20260317 21:19:27.028272 7261 leader_election.cc:343] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:27.029836 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4737071c2aec4449a05bead6af7ad32b" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:27.030400 7322 raft_consensus.cc:3060] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:27.031288 8706 leader_election.cc:290] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:27.038978 7329 raft_consensus.cc:2468] T 70472ad137f34237a5a67f74c94aa36b P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:27.037725 7163 tablet_service.cc:1511] Processing CreateTablet for tablet 3180d86564c945218ec4ac5b35634047 (DEFAULT_TABLE table=default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 [id=54eed51395b74f46b69a56cfa73f3724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.040357 7119 leader_election.cc:304] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.045975 8706 raft_consensus.cc:2804] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:27.046783 8706 raft_consensus.cc:697] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:27.049082 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 3180d86564c945218ec4ac5b35634047 (DEFAULT_TABLE table=default.Table_6ec3da8e67c5441c8f8d0e5c4a12a541 [id=54eed51395b74f46b69a56cfa73f3724]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.050239 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3180d86564c945218ec4ac5b35634047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.054220 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de1582e47f948cfa6c3c0c997ea573f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.057380 7163 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3180d86564c945218ec4ac5b35634047. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.070755 7154 tablet_service.cc:1511] Processing CreateTablet for tablet 74d445b420134ef5aef51f9c8e0f55da (DEFAULT_TABLE table=default.Table_cafc44dd71714a528bda6622dac7997e [id=c11557d0407d4bf08dac964b24545e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.072000 7154 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d445b420134ef5aef51f9c8e0f55da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.085750 7322 raft_consensus.cc:2468] T 4737071c2aec4449a05bead6af7ad32b P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:27.087133 7119 leader_election.cc:304] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.087949 8783 raft_consensus.cc:2804] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:27.088363 8783 raft_consensus.cc:697] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:27.089208 8783 consensus_queue.cc:237] T 4737071c2aec4449a05bead6af7ad32b P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.093096 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 74d445b420134ef5aef51f9c8e0f55da (DEFAULT_TABLE table=default.Table_cafc44dd71714a528bda6622dac7997e [id=c11557d0407d4bf08dac964b24545e44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.094736 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74d445b420134ef5aef51f9c8e0f55da. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.100186 8620 raft_consensus.cc:493] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.100804 8620 raft_consensus.cc:515] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.104162 8620 leader_election.cc:290] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:27.109064 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae1b2a549af6485ca3faee3f0ad845a2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:27.109833 7322 raft_consensus.cc:2468] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:27.111081 7119 leader_election.cc:304] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.120400 7462 raft_consensus.cc:359] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.121107 7462 raft_consensus.cc:385] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.121352 7462 raft_consensus.cc:740] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.121518 8620 raft_consensus.cc:2804] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:27.121860 8620 raft_consensus.cc:493] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:27.047744 8706 consensus_queue.cc:237] T 70472ad137f34237a5a67f74c94aa36b P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.122135 8620 raft_consensus.cc:3060] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:27.130736 8783 raft_consensus.cc:493] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.131325 8783 raft_consensus.cc:515] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:27.134074 6736 catalog_manager.cc:4784] Async tablet task AlterTable RPC for tablet bc2cd63423f844aea29ed3fd839f083e (table default.Table_0475d441c7644d2388e6717057473374 [id=99a93aed10d2424087ea303d139653c2], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bc2cd63423f844aea29ed3fd839f083e consensus state has no leader
I20260317 21:19:27.136436 8783 leader_election.cc:290] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:27.140282 7164 tablet_service.cc:1511] Processing CreateTablet for tablet cbef3d2247dd4b3ba7e1a2495403d8ef (DEFAULT_TABLE table=default.Table_61b756f1959b46f393c2a726704e78aa [id=520224705f0d45feb82bb3a96db2cca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.141570 7164 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef3d2247dd4b3ba7e1a2495403d8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.140455 7300 tablet_service.cc:1511] Processing CreateTablet for tablet cbef3d2247dd4b3ba7e1a2495403d8ef (DEFAULT_TABLE table=default.Table_61b756f1959b46f393c2a726704e78aa [id=520224705f0d45feb82bb3a96db2cca0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.143188 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f69403b3b811419e87916a381656e1ec" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4" is_pre_election: true
I20260317 21:19:27.143805 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbef3d2247dd4b3ba7e1a2495403d8ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.143834 7322 raft_consensus.cc:2468] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 0.
I20260317 21:19:27.159592 7119 leader_election.cc:304] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.162257 8783 raft_consensus.cc:2804] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:27.162756 8783 raft_consensus.cc:493] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:27.163142 8783 raft_consensus.cc:3060] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:27.172794 8620 raft_consensus.cc:515] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.176892 7322 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae1b2a549af6485ca3faee3f0ad845a2" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:27.177594 7322 raft_consensus.cc:3060] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:27.179952 7467 tablet_bootstrap.cc:492] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:27.180397 7467 ts_tablet_manager.cc:1403] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.202s user 0.014s sys 0.020s
I20260317 21:19:27.188230 7462 consensus_queue.cc:260] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.192000 7462 ts_tablet_manager.cc:1434] T aa814dcc2499482eaf0afc5d95bb40f1 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.165s user 0.008s sys 0.000s
I20260317 21:19:27.201236 8620 leader_election.cc:290] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:27.183226 7467 raft_consensus.cc:359] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.221349 7467 raft_consensus.cc:385] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.221699 7467 raft_consensus.cc:740] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.222364 7462 tablet_bootstrap.cc:492] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:27.222288 7467 consensus_queue.cc:260] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.227982 7462 tablet_bootstrap.cc:654] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.232550 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 983d205a22b04c8fa5ae45849fc583ab (DEFAULT_TABLE table=default.Table_db13c9846b0049dca4c784a97c434893 [id=305ca4adf2674d4c8cfa2b63e2316871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.233654 7159 tablet_service.cc:1511] Processing CreateTablet for tablet edd09d2a9ba741bc94ac682018337a83 (DEFAULT_TABLE table=default.Table_d821b351490e4553b71a09c4055e26cd [id=87c45aeb0b21447bac6301a6b5845376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.234710 7159 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd09d2a9ba741bc94ac682018337a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.235321 7322 raft_consensus.cc:2468] T ae1b2a549af6485ca3faee3f0ad845a2 P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:27.235857 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983d205a22b04c8fa5ae45849fc583ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.237896 8818 raft_consensus.cc:493] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.238364 8818 raft_consensus.cc:515] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.239167 7467 ts_tablet_manager.cc:1434] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a: Time spent starting tablet: real 0.058s user 0.005s sys 0.001s
I20260317 21:19:27.240298 7467 tablet_bootstrap.cc:492] T 6ab5f3dc7836425d917bd09f1e6bfe90 P c0ef92fef382401a9564c17e0c223f6a: Bootstrap starting.
I20260317 21:19:27.240357 8818 leader_election.cc:290] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:27.246682 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4495961a559549629dedf8ce032abc9b" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:27.247287 7184 raft_consensus.cc:2468] T 4495961a559549629dedf8ce032abc9b P c0ef92fef382401a9564c17e0c223f6a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f539d1a4a584b8ca83928be49369bf4 in term 0.
I20260317 21:19:27.249627 8818 raft_consensus.cc:493] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.250027 7162 tablet_service.cc:1511] Processing CreateTablet for tablet 983d205a22b04c8fa5ae45849fc583ab (DEFAULT_TABLE table=default.Table_db13c9846b0049dca4c784a97c434893 [id=305ca4adf2674d4c8cfa2b63e2316871]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.250044 8818 raft_consensus.cc:515] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.251042 7162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 983d205a22b04c8fa5ae45849fc583ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.252081 7261 leader_election.cc:304] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.252911 8854 raft_consensus.cc:2804] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:27.253264 8854 raft_consensus.cc:493] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:27.253170 7184 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "69801721e2b7422eba08becf2bcecfa2" candidate_uuid: "2f539d1a4a584b8ca83928be49369bf4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c0ef92fef382401a9564c17e0c223f6a" is_pre_election: true
I20260317 21:19:27.253582 8854 raft_consensus.cc:3060] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:27.256989 7261 leader_election.cc:343] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:19:27.258392 7467 tablet_bootstrap.cc:654] T 6ab5f3dc7836425d917bd09f1e6bfe90 P c0ef92fef382401a9564c17e0c223f6a: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.261097 8818 leader_election.cc:290] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:27.263391 8854 raft_consensus.cc:515] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.265012 8783 raft_consensus.cc:515] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.266846 8783 leader_election.cc:290] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Requested vote from peers d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199), 2f539d1a4a584b8ca83928be49369bf4 (127.6.122.67:32877)
I20260317 21:19:27.271193 7329 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f69403b3b811419e87916a381656e1ec" candidate_uuid: "c0ef92fef382401a9564c17e0c223f6a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2f539d1a4a584b8ca83928be49369bf4"
I20260317 21:19:27.271811 7329 raft_consensus.cc:3060] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:27.277310 8854 leader_election.cc:290] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: Requested vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:27.285005 7329 raft_consensus.cc:2468] T f69403b3b811419e87916a381656e1ec P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c0ef92fef382401a9564c17e0c223f6a in term 1.
I20260317 21:19:27.288134 7302 tablet_service.cc:1511] Processing CreateTablet for tablet d0769fc5106f441ba7060ed4fa681316 (DEFAULT_TABLE table=default.Table_2cd62a2d2daa4868a58114c426bca181 [id=ab3da0a6fea343d3b2d399b3437fe6f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.289299 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d0769fc5106f441ba7060ed4fa681316. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.300096 7462 tablet_bootstrap.cc:492] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:27.300734 7462 ts_tablet_manager.cc:1403] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.079s user 0.012s sys 0.000s
I20260317 21:19:27.302444 7119 leader_election.cc:304] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.308976 8854 raft_consensus.cc:493] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.305222 7462 raft_consensus.cc:359] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.309603 8854 raft_consensus.cc:515] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.309613 7462 raft_consensus.cc:385] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.312574 8854 leader_election.cc:290] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:27.303834 8874 raft_consensus.cc:2804] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:27.313118 8874 raft_consensus.cc:697] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:27.313659 7298 tablet_service.cc:1511] Processing CreateTablet for tablet bdea6939a3e64a279acef3b987e359a3 (DEFAULT_TABLE table=default.Table_39092d36aadb42628dabdb2bf48c1950 [id=831b6f16d49a402e8cc3d53234e050ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.315145 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bdea6939a3e64a279acef3b987e359a3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.315541 8874 consensus_queue.cc:237] T ae1b2a549af6485ca3faee3f0ad845a2 P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.318259 7467 tablet_bootstrap.cc:492] T 6ab5f3dc7836425d917bd09f1e6bfe90 P c0ef92fef382401a9564c17e0c223f6a: No bootstrap required, opened a new log
I20260317 21:19:27.312161 7462 raft_consensus.cc:740] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.318691 7467 ts_tablet_manager.cc:1403] T 6ab5f3dc7836425d917bd09f1e6bfe90 P c0ef92fef382401a9564c17e0c223f6a: Time spent bootstrapping tablet: real 0.079s user 0.003s sys 0.006s
I20260317 21:19:27.303850 7119 leader_election.cc:304] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4, c0ef92fef382401a9564c17e0c223f6a; no voters:
I20260317 21:19:27.320041 7462 consensus_queue.cc:260] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.322683 8783 raft_consensus.cc:2804] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:19:27.323174 8783 raft_consensus.cc:697] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [term 1 LEADER]: Becoming Leader. State: Replica: c0ef92fef382401a9564c17e0c223f6a, State: Running, Role: LEADER
I20260317 21:19:27.328217 8783 consensus_queue.cc:237] T f69403b3b811419e87916a381656e1ec P c0ef92fef382401a9564c17e0c223f6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.332298 7163 tablet_service.cc:1511] Processing CreateTablet for tablet d0769fc5106f441ba7060ed4fa681316 (DEFAULT_TABLE table=default.Table_2cd62a2d2daa4868a58114c426bca181 [id=ab3da0a6fea343d3b2d399b3437fe6f1]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:27.332949 6737 connection.cc:537] client connection to 127.6.122.65:44199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:19:27.333277 6737 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107) [suppressed 1 similar messages]
I20260317 21:19:27.340873 7462 ts_tablet_manager.cc:1434] T 5e080308f93448138a4846ab6bec7dfa P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.040s user 0.004s sys 0.002s
W20260317 21:19:27.358448 6661 connection.cc:537] server connection from 127.6.122.65:49239 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:19:27.357883 7120 connection.cc:537] client connection to 127.6.122.65:44199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:19:27.360966 7120 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:19:27.361198 7462 tablet_bootstrap.cc:492] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
W20260317 21:19:27.365046 6734 connection.cc:537] server connection from 127.6.122.65:33613 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:19:27.369434 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 7090
I20260317 21:19:27.370935 7304 tablet_service.cc:1511] Processing CreateTablet for tablet edd09d2a9ba741bc94ac682018337a83 (DEFAULT_TABLE table=default.Table_d821b351490e4553b71a09c4055e26cd [id=87c45aeb0b21447bac6301a6b5845376]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.329021 7299 tablet_service.cc:1511] Processing CreateTablet for tablet 611ea189ce5d4d3c8c93c18bfd6508e7 (DEFAULT_TABLE table=default.Table_c928476e679c41589639228020653020 [id=25a841c1c6e2466eb130c7fac5dd1e68]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:27.357016 7261 connection.cc:537] server connection from 127.6.122.65:58243 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:19:27.357484 8854 raft_consensus.cc:493] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:19:27.357542 7264 connection.cc:537] client connection to 127.6.122.65:44199 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:19:27.372587 7264 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:19:27.372583 8854 raft_consensus.cc:515] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.375771 8854 leader_election.cc:290] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:27.377100 6737 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.CreateTablet -> {remote=127.6.122.65:44199, user_credentials={real_user=slave}} blocked reactor thread for 43818.7us
I20260317 21:19:27.387740 7302 tablet_service.cc:1511] Processing CreateTablet for tablet 8c214dfdafc9464ea32fffd8223ae990 (DEFAULT_TABLE table=default.Table_bcc32ab4cc0449e0a8c5c696d3c4a193 [id=121b3b215a0b4914a3f18167636ce46e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.372318 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edd09d2a9ba741bc94ac682018337a83. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.390887 7299 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 611ea189ce5d4d3c8c93c18bfd6508e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.391815 7462 tablet_bootstrap.cc:654] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.400408 8881 raft_consensus.cc:493] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:27.401034 8881 raft_consensus.cc:515] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
I20260317 21:19:27.405285 8881 leader_election.cc:290] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 31726fd7f40442e49b2fe0f880793206 (127.6.122.126:40107), 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779)
I20260317 21:19:27.406392 6685 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 11 candidate_status { last_received { term: 10 index: 431 } } ignore_live_leader: false dest_uuid: "31726fd7f40442e49b2fe0f880793206" is_pre_election: true
I20260317 21:19:27.408303 7300 tablet_service.cc:1511] Processing CreateTablet for tablet 613e735b0f414ddf8888be0dfd23ada3 (DEFAULT_TABLE table=default.Table_147e7f28d4344faf9b6289f6c999b6a9 [id=fd0410563367415d8e73a23c52e8e839]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.409492 6751 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" candidate_term: 11 candidate_status { last_received { term: 10 index: 431 } } ignore_live_leader: false dest_uuid: "21186091f2c9410ea6e585c5ef2c6291" is_pre_election: true
I20260317 21:19:27.410950 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [CANDIDATE]: Term 11 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 47e1c983d0224428a9a7b147a0e5e4ef; no voters: 21186091f2c9410ea6e585c5ef2c6291, 31726fd7f40442e49b2fe0f880793206
W20260317 21:19:27.417035 6737 catalog_manager.cc:4729] TS d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7fc38947219047b5abdf71af5554a987: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.424849 8881 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 47e1c983d0224428a9a7b147a0e5e4ef [term 10 FOLLOWER]: Leader pre-election lost for term 11. Reason: could not achieve majority
I20260317 21:19:27.425935 7305 tablet_service.cc:1511] Processing CreateTablet for tablet 19d415c5f9cd4d01a9170ae391e22d70 (DEFAULT_TABLE table=default.Table_1f45d1a0799640caab2984b6590f965e [id=21da450e94ef40b5aa0c9d5e3e8a7b5b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.433334 7302 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8c214dfdafc9464ea32fffd8223ae990. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.439265 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 613e735b0f414ddf8888be0dfd23ada3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.444242 7462 tablet_bootstrap.cc:492] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:27.444751 7462 ts_tablet_manager.cc:1403] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.084s user 0.009s sys 0.004s
I20260317 21:19:27.462530 7462 raft_consensus.cc:359] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.463312 7462 raft_consensus.cc:385] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.463577 7462 raft_consensus.cc:740] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.464303 7462 consensus_queue.cc:260] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
W20260317 21:19:27.448441 7264 leader_election.cc:336] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.467319 7462 ts_tablet_manager.cc:1434] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.022s user 0.006s sys 0.000s
I20260317 21:19:27.465616 7264 leader_election.cc:304] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.468288 8854 raft_consensus.cc:2749] T eb3a61ed542741d88d89ef6b4c53e9e6 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.469348 7264 leader_election.cc:336] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.470328 7305 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19d415c5f9cd4d01a9170ae391e22d70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.469717 7264 leader_election.cc:304] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.476881 8854 raft_consensus.cc:2749] T f5027b892229491eb5a829cacd29a700 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.477873 7264 consensus_peers.cc:597] T a37e53e326914575830e30915ad69cc7 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:27.486071 7298 tablet_service.cc:1511] Processing CreateTablet for tablet 54e14c247b10477fafc6ccd6a5be3761 (DEFAULT_TABLE table=default.Table_15292a1c98be431b9d3cacbc7e767c75 [id=0f32cb4e70be48c2a548e9e2d4805822]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.487344 7298 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54e14c247b10477fafc6ccd6a5be3761. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:19:27.478322 7264 consensus_peers.cc:597] T 740ae685629041f39bbbc8ce715d662e P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.493099 7264 leader_election.cc:336] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.493724 7264 leader_election.cc:304] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.494500 8854 raft_consensus.cc:2749] T 17610bdee98f49c4b1d8b5481867c6e8 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.495590 7264 consensus_peers.cc:597] T a4cee7952cf5428b98340789a48d2da2 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.496174 7264 consensus_peers.cc:597] T bc2cd63423f844aea29ed3fd839f083e P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.496582 7264 consensus_peers.cc:597] T 2a221dfad830428db2329d6e543f41dc P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.497061 7264 leader_election.cc:336] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.497568 7264 leader_election.cc:304] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.500339 8854 raft_consensus.cc:2749] T 6ab5f3dc7836425d917bd09f1e6bfe90 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.501430 7264 consensus_peers.cc:597] T b2dccb915fad4666bd7978f8c416db44 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.505978 7264 consensus_peers.cc:597] T d05973b63a8c445995ca51d8aec7bcf9 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.506652 7264 consensus_peers.cc:597] T 252f728d183f440a9ef42894508d5755 P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.508787 7264 leader_election.cc:336] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.509517 7462 tablet_bootstrap.cc:492] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:27.523864 7462 tablet_bootstrap.cc:654] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.526875 7264 leader_election.cc:304] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.532974 8820 raft_consensus.cc:2749] T 646a115e133b43eebc1674a862ab7838 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.534310 7264 consensus_peers.cc:597] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260317 21:19:27.534837 7264 leader_election.cc:336] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
W20260317 21:19:27.541078 7264 leader_election.cc:336] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.541875 7264 leader_election.cc:304] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.537694 7304 tablet_service.cc:1511] Processing CreateTablet for tablet 47ce1e486be64aa2803430a33c780348 (DEFAULT_TABLE table=default.Table_bcc665be09a84bbbb32857f85e717c0d [id=b0298168ff864e988c6c9d94351fd33e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:19:27.543160 8820 raft_consensus.cc:2749] T 69801721e2b7422eba08becf2bcecfa2 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.544034 7264 leader_election.cc:336] T 4495961a559549629dedf8ce032abc9b P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
W20260317 21:19:27.545025 7264 leader_election.cc:336] T 7b7b54cbff0c4fd5813d197ddbc899cc P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
W20260317 21:19:27.555475 7264 leader_election.cc:336] T 2152e5cbd2ce4159b0901ba2ffe41f40 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.562638 7462 tablet_bootstrap.cc:492] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
W20260317 21:19:27.562661 7264 consensus_peers.cc:597] T f3b5825c77ac4d69a39978932051db8c P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:27.563082 7300 tablet_service.cc:1511] Processing CreateTablet for tablet ad8163635c344d29b8aaa2275539bd22 (DEFAULT_TABLE table=default.Table_1342223726fb480aba4932d06e045aaf [id=06e60ad198054f26b5512c8f53e70753]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:19:27.563455 7264 leader_election.cc:336] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.563104 7462 ts_tablet_manager.cc:1403] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.003s
I20260317 21:19:27.563995 7264 leader_election.cc:304] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.567530 8820 raft_consensus.cc:2749] T 32be4818a67a41c998f04d7cedf7356e P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:19:27.567720 8812 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
W20260317 21:19:27.568504 7264 leader_election.cc:336] T 2607b9978c32466eb6dc8b8acfb429ce P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
W20260317 21:19:27.570652 7264 leader_election.cc:336] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.571105 7264 leader_election.cc:304] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2f539d1a4a584b8ca83928be49369bf4; no voters: c0ef92fef382401a9564c17e0c223f6a, d3cb3b49e3e94537a6e434ea4e037c3f
I20260317 21:19:27.566157 7462 raft_consensus.cc:359] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.578699 7304 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 47ce1e486be64aa2803430a33c780348. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.579545 7462 raft_consensus.cc:385] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.579810 7462 raft_consensus.cc:740] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.580441 7462 consensus_queue.cc:260] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.581408 8820 raft_consensus.cc:2749] T 6cf1a0922ed54674aaff8b0f49751c13 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:19:27.582278 7264 consensus_peers.cc:597] T a4d64c48641b484ab443380808544d2f P 2f539d1a4a584b8ca83928be49369bf4 -> Peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Couldn't send request to peer d3cb3b49e3e94537a6e434ea4e037c3f. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:19:27.589898 7462 ts_tablet_manager.cc:1434] T dde40a1c77854adcabc2fff3baaf4c81 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.026s user 0.005s sys 0.000s
I20260317 21:19:27.590628 7300 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad8163635c344d29b8aaa2275539bd22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:27.610998 8820 raft_consensus.cc:493] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.611515 8820 raft_consensus.cc:515] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.613453 8820 leader_election.cc:290] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
W20260317 21:19:27.618870 7264 leader_election.cc:336] T 1c3beb9232d74e45ab713ae8241eec4c P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.626219 7462 tablet_bootstrap.cc:492] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:27.633874 7462 tablet_bootstrap.cc:654] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.679543 8890 raft_consensus.cc:493] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Starting pre-election (detected failure of leader 21186091f2c9410ea6e585c5ef2c6291)
I20260317 21:19:27.680135 7462 tablet_bootstrap.cc:492] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:27.680549 7462 ts_tablet_manager.cc:1403] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.055s user 0.010s sys 0.004s
I20260317 21:19:27.683079 8820 raft_consensus.cc:493] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:27.683499 8820 raft_consensus.cc:515] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.685377 8820 leader_election.cc:290] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c0ef92fef382401a9564c17e0c223f6a (127.6.122.66:45979), d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199)
I20260317 21:19:27.685633 8890 raft_consensus.cc:515] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [term 10 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31726fd7f40442e49b2fe0f880793206" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 40107 } } peers { permanent_uuid: "21186091f2c9410ea6e585c5ef2c6291" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 32779 } } peers { permanent_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 38857 } }
W20260317 21:19:27.691162 7264 leader_election.cc:336] T 9c3b084b04e944d7b919ec1965742e9f P 2f539d1a4a584b8ca83928be49369bf4 [CANDIDATE]: Term 1 pre-election: RPC error from VoteRequest() call to peer d3cb3b49e3e94537a6e434ea4e037c3f (127.6.122.65:44199): Network error: Client connection negotiation failed: client connection to 127.6.122.65:44199: connect: Connection refused (error 111)
I20260317 21:19:27.691879 7462 raft_consensus.cc:359] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.693136 7462 raft_consensus.cc:385] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.693813 7462 raft_consensus.cc:740] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.697276 7918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "31726fd7f40442e49b2fe0f880793206" candidate_term: 11 candidate_status { last_received { term: 10 index: 431 } } ignore_live_leader: false dest_uuid: "47e1c983d0224428a9a7b147a0e5e4ef" is_pre_election: true
I20260317 21:19:27.698163 7462 consensus_queue.cc:260] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.709416 8890 leader_election.cc:290] T 00000000000000000000000000000000 P 31726fd7f40442e49b2fe0f880793206 [CANDIDATE]: Term 11 pre-election: Requested pre-vote from peers 21186091f2c9410ea6e585c5ef2c6291 (127.6.122.125:32779), 47e1c983d0224428a9a7b147a0e5e4ef (127.6.122.124:38857)
I20260317 21:19:27.736774 7462 ts_tablet_manager.cc:1434] T d8247f6d37624ca3afcf0702eb6bdd55 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.056s user 0.008s sys 0.000s
I20260317 21:19:27.737828 7462 tablet_bootstrap.cc:492] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:27.760342 7462 tablet_bootstrap.cc:654] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:27.811515 7462 tablet_bootstrap.cc:492] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4: No bootstrap required, opened a new log
I20260317 21:19:27.811976 7462 ts_tablet_manager.cc:1403] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent bootstrapping tablet: real 0.074s user 0.010s sys 0.000s
I20260317 21:19:27.814502 7462 raft_consensus.cc:359] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.815225 7462 raft_consensus.cc:385] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:27.815510 7462 raft_consensus.cc:740] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f539d1a4a584b8ca83928be49369bf4, State: Initialized, Role: FOLLOWER
I20260317 21:19:27.816210 7462 consensus_queue.cc:260] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3cb3b49e3e94537a6e434ea4e037c3f" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 44199 } } peers { permanent_uuid: "c0ef92fef382401a9564c17e0c223f6a" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 45979 } } peers { permanent_uuid: "2f539d1a4a584b8ca83928be49369bf4" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 32877 } }
I20260317 21:19:27.827505 7462 ts_tablet_manager.cc:1434] T 9a1819f74297406e8ebb28ae847abdd3 P 2f539d1a4a584b8ca83928be49369bf4: Time spent starting tablet: real 0.015s user 0.002s sys 0.003s
I20260317 21:19:27.831897 7462 tablet_bootstrap.cc:492] T 43702e625cb147339bd5871a5ee83e17 P 2f539d1a4a584b8ca83928be49369bf4: Bootstrap starting.
I20260317 21:19:27.870105 7462 tablet_bootstrap.cc:654] T 43702e625cb147339bd5871a5ee83e17 P 2f539d1a4a584b8ca83928be49369bf4: Neither blocks nor log segments found. Creating new log.
W20260317 21:19:27.871196 6734 connection.cc:537] client connection to 127.6.122.66:45979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:19:27.871474 7261 connection.cc:537] client connection to 127.6.122.66:45979 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:19:27.872958 6659 connection.cc:537] server connection from 127.6.122.66:45181 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:19:27.883519 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 7238
W20260317 21:19:27.997432 6661 connection.cc:537] server connection from 127.6.122.67:36209 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:19:28.007941 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 6642
I20260317 21:19:28.081341 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 6712
I20260317 21:19:28.156385 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 7443
2026-03-17T21:19:28Z chronyd exiting
I20260317 21:19:28.221554 6633 master-stress-test.cc:585] Tables created: 188
I20260317 21:19:28.221791 6633 master-stress-test.cc:586] Tables altered: 85
I20260317 21:19:28.221975 6633 master-stress-test.cc:587] Tables deleted: 66
I20260317 21:19:28.222133 6633 master-stress-test.cc:588] Tablets replaced: 40
I20260317 21:19:28.222298 6633 master-stress-test.cc:589] Masters restarted: 1
[ OK ] CatalogManagerConfigurations/MasterStressTest.Test/0 (57963 ms)
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/3
2026-03-17T21:19:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-03-17T21:19:28Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklBwffU/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklBwffU/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/security.properties
2026-03-17 21:19:37: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklBwffU/thirdparty/src/hive-3.1.2/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-tasklBwffU/thirdparty/src/hadoop-3.4.1/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260317 21:19:48.491842 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.126:32989
--webserver_interface=127.6.122.126
--webserver_port=0
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--hive_metastore_uris=thrift://127.0.0.1:42187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20260317 21:19:48.832984 9176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:19:48.833673 9176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:19:48.834185 9176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:19:48.870157 9176 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260317 21:19:48.870436 9176 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260317 21:19:48.870738 9176 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:19:48.870988 9176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:19:48.871275 9176 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:19:48.871531 9176 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:19:48.909801 9176 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.126:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.122.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:19:48.911758 9176 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:19:48.913816 9176 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:19:48.927304 9182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:48.927507 9183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:50.205333 9185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:50.208961 9184 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1280 milliseconds
W20260317 21:19:50.209604 9176 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.282s user 0.473s sys 0.805s
W20260317 21:19:50.210039 9176 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.283s user 0.473s sys 0.805s
I20260317 21:19:50.210331 9176 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:19:50.211478 9176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:19:50.214218 9176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:19:50.215675 9176 hybrid_clock.cc:648] HybridClock initialized: now 1773782390215629 us; error 65 us; skew 500 ppm
I20260317 21:19:50.216566 9176 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:19:50.225327 9176 webserver.cc:492] Webserver started at http://127.6.122.126:43267/ using document root <none> and password file <none>
I20260317 21:19:50.226393 9176 fs_manager.cc:362] Metadata directory not provided
I20260317 21:19:50.226691 9176 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:19:50.227203 9176 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:19:50.232060 9176 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/instance:
uuid: "d6b9687775954282a11f2cf829d24bac"
format_stamp: "Formatted at 2026-03-17 21:19:50 on dist-test-slave-979h"
I20260317 21:19:50.233431 9176 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal/instance:
uuid: "d6b9687775954282a11f2cf829d24bac"
format_stamp: "Formatted at 2026-03-17 21:19:50 on dist-test-slave-979h"
I20260317 21:19:50.242528 9176 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260317 21:19:50.249347 9192 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:50.250766 9176 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260317 21:19:50.251186 9176 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
uuid: "d6b9687775954282a11f2cf829d24bac"
format_stamp: "Formatted at 2026-03-17 21:19:50 on dist-test-slave-979h"
I20260317 21:19:50.251585 9176 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:19:50.335994 9176 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:19:50.337895 9176 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:19:50.338488 9176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:19:50.413379 9176 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.126:32989
I20260317 21:19:50.413476 9243 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.126:32989 every 8 connection(s)
I20260317 21:19:50.416253 9176 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
I20260317 21:19:50.420426 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9176
I20260317 21:19:50.421149 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal/instance
I20260317 21:19:50.424147 9244 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:50.425067 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.125:38261
--webserver_interface=127.6.122.125
--webserver_port=0
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--hive_metastore_uris=thrift://127.0.0.1:42187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260317 21:19:50.440251 9244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:50.474577 9244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } has no permanent_uuid. Determining permanent_uuid...
W20260317 21:19:50.477975 9195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:19:50.483916 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:19:50.533591 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 1
W20260317 21:19:50.538033 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:19:50.606581 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 2
W20260317 21:19:50.611843 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:19:50.703383 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 3
W20260317 21:19:50.708199 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:19:50.774616 9245 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:19:50.775345 9245 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:19:50.775866 9245 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:19:50.809645 9245 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260317 21:19:50.809952 9245 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260317 21:19:50.810292 9245 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:19:50.810587 9245 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:19:50.810865 9245 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:19:50.811129 9245 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:19:50.848254 9245 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal
--hive_metastore_uris=thrift://127.0.0.1:42187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.125:38261
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.6.122.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:19:50.849754 9245 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:19:50.851768 9245 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260317 21:19:50.851758 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 4
W20260317 21:19:50.856293 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:19:50.864456 9254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:19:51.155846 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 5
W20260317 21:19:51.163270 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:19:51.710978 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 6
W20260317 21:19:51.725852 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:19:50.866111 9255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:52.182274 9257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:52.184988 9256 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1314 milliseconds
I20260317 21:19:52.185137 9245 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:19:52.186462 9245 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:19:52.189096 9245 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:19:52.190491 9245 hybrid_clock.cc:648] HybridClock initialized: now 1773782392190464 us; error 76 us; skew 500 ppm
I20260317 21:19:52.191433 9245 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:19:52.198238 9245 webserver.cc:492] Webserver started at http://127.6.122.125:35089/ using document root <none> and password file <none>
I20260317 21:19:52.199296 9245 fs_manager.cc:362] Metadata directory not provided
I20260317 21:19:52.199558 9245 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:19:52.200064 9245 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:19:52.205196 9245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data/instance:
uuid: "a4719f316c614a9cadaf5edab834804d"
format_stamp: "Formatted at 2026-03-17 21:19:52 on dist-test-slave-979h"
I20260317 21:19:52.206509 9245 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal/instance:
uuid: "a4719f316c614a9cadaf5edab834804d"
format_stamp: "Formatted at 2026-03-17 21:19:52 on dist-test-slave-979h"
I20260317 21:19:52.214196 9245 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260317 21:19:52.220669 9267 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:52.222291 9245 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260317 21:19:52.222764 9245 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal
uuid: "a4719f316c614a9cadaf5edab834804d"
format_stamp: "Formatted at 2026-03-17 21:19:52 on dist-test-slave-979h"
I20260317 21:19:52.223150 9245 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:19:52.280869 9245 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:19:52.282473 9245 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:19:52.282979 9245 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:19:52.360507 9245 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.125:38261
I20260317 21:19:52.360616 9318 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.125:38261 every 8 connection(s)
I20260317 21:19:52.363575 9245 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/data/info.pb
I20260317 21:19:52.367625 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9245
I20260317 21:19:52.368279 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-1/wal/instance
I20260317 21:19:52.369819 9319 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:52.371385 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:41015
--webserver_interface=127.6.122.124
--webserver_port=0
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--hive_metastore_uris=thrift://127.0.0.1:42187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260317 21:19:52.382309 9319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:52.406423 9319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:52.426045 9319 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } has no permanent_uuid. Determining permanent_uuid...
W20260317 21:19:52.429118 9270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
W20260317 21:19:52.434386 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:52.484045 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 1
W20260317 21:19:52.489251 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:52.557888 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 2
W20260317 21:19:52.562767 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:52.654258 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 3
W20260317 21:19:52.658596 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
W20260317 21:19:52.729487 9320 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:19:52.730152 9320 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:19:52.730654 9320 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:19:52.764315 9320 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260317 21:19:52.764662 9320 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260317 21:19:52.764941 9320 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:19:52.765208 9320 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:19:52.765482 9320 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:19:52.765739 9320 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:19:52.786564 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } attempt: 7
I20260317 21:19:52.799335 9244 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:52.802062 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 4
W20260317 21:19:52.805888 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
W20260317 21:19:52.806900 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:52.813014 9320 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.122.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:19:52.814452 9320 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:19:52.816134 9320 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:19:52.828348 9332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:19:52.864461 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 1
W20260317 21:19:52.873312 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:52.955124 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 2
W20260317 21:19:52.962812 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:53.048599 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 3
W20260317 21:19:53.056938 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:53.106293 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 5
W20260317 21:19:53.113901 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:53.197662 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 4
W20260317 21:19:53.204862 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:53.488574 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 5
W20260317 21:19:53.501617 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:53.661630 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 6
W20260317 21:19:53.677945 9319 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:19:54.054244 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 6
W20260317 21:19:54.064384 9244 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.6.122.124:41015: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
W20260317 21:19:52.829690 9333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:52.833962 9335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:19:54.120911 9334 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1288 milliseconds
I20260317 21:19:54.121083 9320 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:19:54.122555 9320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:19:54.125908 9320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:19:54.127394 9320 hybrid_clock.cc:648] HybridClock initialized: now 1773782394127337 us; error 75 us; skew 500 ppm
I20260317 21:19:54.128516 9320 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:19:54.136440 9320 webserver.cc:492] Webserver started at http://127.6.122.124:33267/ using document root <none> and password file <none>
I20260317 21:19:54.137543 9320 fs_manager.cc:362] Metadata directory not provided
I20260317 21:19:54.137817 9320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:19:54.138321 9320 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:19:54.143343 9320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/instance:
uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9"
format_stamp: "Formatted at 2026-03-17 21:19:54 on dist-test-slave-979h"
I20260317 21:19:54.144995 9320 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal/instance:
uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9"
format_stamp: "Formatted at 2026-03-17 21:19:54 on dist-test-slave-979h"
I20260317 21:19:54.153501 9320 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260317 21:19:54.160270 9346 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:54.161768 9320 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260317 21:19:54.162197 9320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9"
format_stamp: "Formatted at 2026-03-17 21:19:54 on dist-test-slave-979h"
I20260317 21:19:54.162621 9320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:19:54.231786 9320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:19:54.233492 9320 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:19:54.233999 9320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:19:54.312736 9320 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.124:41015
I20260317 21:19:54.312808 9397 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.124:41015 every 8 connection(s)
I20260317 21:19:54.315531 9320 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
I20260317 21:19:54.321871 9398 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:19:54.326052 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9320
I20260317 21:19:54.326596 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal/instance
I20260317 21:19:54.338716 9398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:54.371448 9398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:54.385442 9398 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } has no permanent_uuid. Determining permanent_uuid...
I20260317 21:19:54.404779 9398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Bootstrap starting.
I20260317 21:19:54.412588 9398 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:54.414677 9398 log.cc:826] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Log is configured to *not* fsync() on all Append() calls
I20260317 21:19:54.420153 9398 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: No bootstrap required, opened a new log
I20260317 21:19:54.443078 9398 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:54.444034 9398 raft_consensus.cc:385] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:54.444410 9398 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed19cc1cc9e74a0d9b8dc4a609f1bea9, State: Initialized, Role: FOLLOWER
I20260317 21:19:54.445499 9398 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:54.449378 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:54.450259 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:54.451064 9398 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: configured and running, proceeding with master startup.
W20260317 21:19:54.491228 9352 tablet.cc:2379] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260317 21:19:54.498500 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.499027 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.514171 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.514705 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.480262 9320 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: OK, instance_id { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" instance_seqno: 1773782390392934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260317 21:19:54.516614 9320 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:19:54.529392 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.529945 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.544668 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.545120 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.559898 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.560492 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.575803 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.576239 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.591116 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.591499 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.605999 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.606369 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.620947 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.621328 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.637042 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.637408 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.652832 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.653244 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.667861 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.668248 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.682611 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.682972 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.697374 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.697798 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.712603 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.713016 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.727626 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.728014 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.738623 9319 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 7
W20260317 21:19:54.742599 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.743044 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.758824 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.759251 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.761293 9319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Bootstrap starting.
I20260317 21:19:54.767333 9319 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:54.769480 9319 log.cc:826] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Log is configured to *not* fsync() on all Append() calls
W20260317 21:19:54.773705 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.774070 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.774637 9319 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: No bootstrap required, opened a new log
W20260317 21:19:54.788625 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.789106 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.797354 9319 raft_consensus.cc:359] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:54.798090 9319 raft_consensus.cc:385] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:54.798386 9319 raft_consensus.cc:740] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4719f316c614a9cadaf5edab834804d, State: Initialized, Role: FOLLOWER
I20260317 21:19:54.799216 9319 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:54.802246 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:54.803241 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: This master's current role is: FOLLOWER
W20260317 21:19:54.803507 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.803874 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.804463 9319 sys_catalog.cc:565] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: configured and running, proceeding with master startup.
W20260317 21:19:54.818306 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.818912 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.831717 9406 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:19:54.832796 9406 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
W20260317 21:19:54.834272 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.834666 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.839350 9294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:19:54.840345 9294 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ed19cc1cc9e74a0d9b8dc4a609f1bea9 in term 0.
I20260317 21:19:54.841171 9406 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
I20260317 21:19:54.841848 9349 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters:
I20260317 21:19:54.842885 9406 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:19:54.843269 9406 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:19:54.843652 9406 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:54.851645 9406 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:54.858589 9406 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 1 election: Requested vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
I20260317 21:19:54.859984 9294 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d"
I20260317 21:19:54.860957 9294 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 0 FOLLOWER]: Advancing to term 1
W20260317 21:19:54.867524 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.867875 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.870357 9440 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260317 21:19:54.876699 9294 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ed19cc1cc9e74a0d9b8dc4a609f1bea9 in term 1.
I20260317 21:19:54.878860 9349 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters:
I20260317 21:19:54.879719 9406 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 FOLLOWER]: Leader election won for term 1
W20260317 21:19:54.883736 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.884109 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.890057 9406 raft_consensus.cc:697] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 LEADER]: Becoming Leader. State: Replica: ed19cc1cc9e74a0d9b8dc4a609f1bea9, State: Running, Role: LEADER
I20260317 21:19:54.891188 9406 consensus_queue.cc:237] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:54.906800 9441 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed19cc1cc9e74a0d9b8dc4a609f1bea9. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:54.907872 9441 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: LEADER
W20260317 21:19:54.914431 9420 catalog_manager.cc:1568] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.914974 9420 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260317 21:19:54.916997 9440 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.917984 9440 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.919708 9444 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:19:54.927210 9444 catalog_manager.cc:1494] Initializing Kudu cluster ID...
W20260317 21:19:54.932855 9440 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.933358 9440 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.874548 9245 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: OK, instance_id { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" instance_seqno: 1773782390392934 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260317 21:19:54.939441 9245 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:19:54.947666 9440 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260317 21:19:54.948202 9440 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:54.951107 9294 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Refusing update from remote peer ed19cc1cc9e74a0d9b8dc4a609f1bea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:19:54.953398 9441 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:54.989854 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed19cc1cc9e74a0d9b8dc4a609f1bea9. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:54.996791 9445 mvcc.cc:204] Tried to move back new op lower bound from 7265412689698115584 to 7265412689485082624. Current Snapshot: MvccSnapshot[applied={T|T < 7265412689698115584}]
W20260317 21:19:55.000016 9440 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260317 21:19:55.000849 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: This master's current role is: FOLLOWER
W20260317 21:19:55.001529 9440 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:55.004434 9406 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:55.005231 9406 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:55.007890 9444 catalog_manager.cc:1357] Generated new cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:19:55.008244 9444 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:19:55.021512 9425 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:55.022316 9425 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:55.030608 9440 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:19:55.031036 9440 catalog_manager.cc:1562] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: loading cluster ID for follower catalog manager: success
W20260317 21:19:55.037230 9440 catalog_manager.cc:1586] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260317 21:19:55.037717 9440 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:55.045766 9444 catalog_manager.cc:1380] Generated new certificate authority record
I20260317 21:19:55.048050 9444 catalog_manager.cc:1514] Loading token signing keys...
W20260317 21:19:55.075570 9440 catalog_manager.cc:1586] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: acquiring CA information for follower catalog manager: Not found: root CA entry not found
W20260317 21:19:55.076782 9440 catalog_manager.cc:883] Not found: root CA entry not found: failed to prepare follower catalog manager, will retry
I20260317 21:19:55.087688 9444 catalog_manager.cc:6044] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Generated new TSK 0
I20260317 21:19:55.091912 9444 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:19:55.097873 9244 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } attempt: 7
I20260317 21:19:55.108702 9440 catalog_manager.cc:1584] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: acquiring CA information for follower catalog manager: success
I20260317 21:19:55.115756 9440 catalog_manager.cc:1612] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260317 21:19:55.117858 9444 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260317 21:19:55.122208 9444 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260317 21:19:55.136034 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Bootstrap starting.
I20260317 21:19:55.144716 9244 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Neither blocks nor log segments found. Creating new log.
I20260317 21:19:55.153995 9244 log.cc:826] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Log is configured to *not* fsync() on all Append() calls
I20260317 21:19:55.157899 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.122.65:0
--local_ip_for_outbound_sockets=127.6.122.65
--webserver_interface=127.6.122.65
--webserver_port=0
--tserver_master_addrs=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260317 21:19:55.165644 9244 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: No bootstrap required, opened a new log
I20260317 21:19:55.212865 9244 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:55.213800 9244 raft_consensus.cc:385] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:19:55.214152 9244 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b9687775954282a11f2cf829d24bac, State: Initialized, Role: FOLLOWER
I20260317 21:19:55.215258 9244 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:55.220242 9244 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:19:55.221612 9217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:19:55.222821 9218 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac"
I20260317 21:19:55.220474 9219 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:55.233981 9217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Refusing update from remote peer ed19cc1cc9e74a0d9b8dc4a609f1bea9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260317 21:19:55.234659 9454 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:55.238442 9441 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:19:55.238773 9454 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:55.309639 9454 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader ed19cc1cc9e74a0d9b8dc4a609f1bea9. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:55.317205 9454 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:55.316311 9464 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:55.317888 9464 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:19:55.447336 9474 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:19:55.455983 9474 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: loading cluster ID for follower catalog manager: success
I20260317 21:19:55.477788 9474 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: acquiring CA information for follower catalog manager: success
I20260317 21:19:55.492733 9474 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260317 21:19:55.609516 9476 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:55.618105 9476 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:55.627537 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:55.630082 9476 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:55.631320 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:55.633386 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:55.636479 9476 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:55.738333 9441 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:19:55.739271 9441 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: LEADER
I20260317 21:19:56.215724 9484 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:56.216338 9484 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:56.219360 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:56.221815 9484 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:56.223001 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:56.224625 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
W20260317 21:19:56.225708 9453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260317 21:19:56.226086 9484 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:19:56.226392 9453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:19:56.227531 9453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:19:56.290649 9453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:19:56.292249 9453 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.122.65
I20260317 21:19:56.373720 9453 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.122.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.6.122.65
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.122.65
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:19:56.375435 9453 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:19:56.377642 9453 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:19:56.433399 9486 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:19:56.740204 9492 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:56.741333 9492 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:56.744760 9492 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:56.747464 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:19:56.749688 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:56.753489 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:56.755218 9492 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:57.196691 9493 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:57.197351 9493 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:57.201418 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:57.203462 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:57.206254 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:57.212798 9493 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:57.216126 9493 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:57.277403 9495 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:57.277920 9495 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:57.279769 9495 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:57.282886 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:19:57.287038 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:57.290131 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:57.291527 9495 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:19:56.446547 9487 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:19:56.446929 9453 server_base.cc:1060] running on GCE node
W20260317 21:19:56.445698 9490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:19:58.098816 9453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:19:58.105836 9453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:19:58.109588 9453 hybrid_clock.cc:648] HybridClock initialized: now 1773782398109506 us; error 82 us; skew 500 ppm
I20260317 21:19:58.111027 9453 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:19:58.121462 9453 webserver.cc:492] Webserver started at http://127.6.122.65:46005/ using document root <none> and password file <none>
I20260317 21:19:58.123023 9453 fs_manager.cc:362] Metadata directory not provided
I20260317 21:19:58.123348 9453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:19:58.123996 9453 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:19:58.131024 9453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data/instance:
uuid: "9b9886349405418f80c5511b013ffbd4"
format_stamp: "Formatted at 2026-03-17 21:19:58 on dist-test-slave-979h"
I20260317 21:19:58.133030 9453 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/instance:
uuid: "9b9886349405418f80c5511b013ffbd4"
format_stamp: "Formatted at 2026-03-17 21:19:58 on dist-test-slave-979h"
I20260317 21:19:58.144493 9453 fs_manager.cc:696] Time spent creating directory manager: real 0.011s user 0.002s sys 0.008s
I20260317 21:19:58.153596 9500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:58.155082 9453 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.003s
I20260317 21:19:58.155530 9453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal
uuid: "9b9886349405418f80c5511b013ffbd4"
format_stamp: "Formatted at 2026-03-17 21:19:58 on dist-test-slave-979h"
I20260317 21:19:58.155992 9453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:19:58.257124 9453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:19:58.259187 9453 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:19:58.259783 9453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:19:58.263899 9453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:19:58.270366 9453 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:19:58.270699 9453 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:58.271013 9453 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:19:58.271229 9453 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:19:58.292850 9510 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:58.294159 9510 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:58.297070 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:19:58.297598 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:58.296219 9510 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:58.299111 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:58.301216 9510 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:58.601469 9585 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:58.602089 9585 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:58.605192 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:58.605337 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:58.607069 9585 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:58.608369 9268 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:58.609983 9585 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:58.665339 9453 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.65:40263
I20260317 21:19:58.665410 9614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.65:40263 every 8 connection(s)
I20260317 21:19:58.669289 9453 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/data/info.pb
I20260317 21:19:58.671626 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9453
I20260317 21:19:58.672220 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/instance
I20260317 21:19:58.685225 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.122.66:0
--local_ip_for_outbound_sockets=127.6.122.66
--webserver_interface=127.6.122.66
--webserver_port=0
--tserver_master_addrs=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260317 21:19:58.767620 9616 heartbeater.cc:344] Connected to a master server at 127.6.122.124:41015
I20260317 21:19:58.768193 9616 heartbeater.cc:461] Registering TS with master...
I20260317 21:19:58.769830 9616 heartbeater.cc:507] Master 127.6.122.124:41015 requested a full tablet report, sending...
I20260317 21:19:58.773398 9363 ts_manager.cc:194] Registered new tserver with Master: 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:19:58.777057 9615 heartbeater.cc:344] Connected to a master server at 127.6.122.125:38261
I20260317 21:19:58.777328 9618 heartbeater.cc:344] Connected to a master server at 127.6.122.126:32989
I20260317 21:19:58.777457 9615 heartbeater.cc:461] Registering TS with master...
I20260317 21:19:58.777798 9618 heartbeater.cc:461] Registering TS with master...
I20260317 21:19:58.778951 9615 heartbeater.cc:507] Master 127.6.122.125:38261 requested a full tablet report, sending...
I20260317 21:19:58.779057 9618 heartbeater.cc:507] Master 127.6.122.126:32989 requested a full tablet report, sending...
I20260317 21:19:58.782481 9284 ts_manager.cc:194] Registered new tserver with Master: 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:19:58.782626 9209 ts_manager.cc:194] Registered new tserver with Master: 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:19:58.776772 9363 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.122.65:57233
W20260317 21:19:59.148624 9619 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:19:59.149271 9619 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:19:59.149904 9619 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:19:59.188611 9619 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:19:59.189731 9619 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.122.66
I20260317 21:19:59.208616 9631 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:59.209483 9631 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:59.212663 9631 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:59.214107 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:59.214097 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:19:59.216058 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:59.216912 9631 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:59.233201 9619 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.122.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.6.122.66
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.122.66
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:19:59.235011 9619 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:19:59.237033 9619 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260317 21:19:59.245599 9632 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:59.246243 9632 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:59.250555 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:59.251921 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:59.255739 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:59.255784 9632 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
W20260317 21:19:59.257645 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:19:59.258036 9632 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:19:59.290712 9616 heartbeater.cc:499] Master 127.6.122.124:41015 was elected leader, sending a full tablet report...
I20260317 21:19:59.790161 9641 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:19:59.792522 9641 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:19:59.815794 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:19:59.827736 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:19:59.831928 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:19:59.840126 9641 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:19:59.841444 9642 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:00.177238 9643 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:00.179545 9643 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:00.186959 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:20:00.195415 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:00.200364 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:00.206990 9643 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:00.208841 9643 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:00.353895 9641 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:00.354728 9641 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:00.359709 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:20:00.366487 9641 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:00.367574 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:00.369037 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:00.370419 9641 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:20:00.658913 9633 debug-util.cc:398] Leaking SignalData structure 0x7b0800002660 after lost signal to thread 9619
W20260317 21:20:00.993495 9619 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.728s user 0.490s sys 1.088s
W20260317 21:20:00.994069 9619 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.729s user 0.490s sys 1.088s
W20260317 21:19:59.266817 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:20:00.995942 9637 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Resolving timed out after 1728 milliseconds
W20260317 21:20:01.005455 9638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:01.005517 9619 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:20:01.006966 9619 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:20:01.010208 9619 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:20:01.013823 9619 hybrid_clock.cc:648] HybridClock initialized: now 1773782401013748 us; error 78 us; skew 500 ppm
I20260317 21:20:01.014864 9619 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:20:01.023595 9619 webserver.cc:492] Webserver started at http://127.6.122.66:40053/ using document root <none> and password file <none>
I20260317 21:20:01.025045 9619 fs_manager.cc:362] Metadata directory not provided
I20260317 21:20:01.025352 9619 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:20:01.025882 9619 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:20:01.031319 9619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data/instance:
uuid: "f738662691924b8a8f9b4b7cf1804923"
format_stamp: "Formatted at 2026-03-17 21:20:01 on dist-test-slave-979h"
I20260317 21:20:01.032878 9619 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/instance:
uuid: "f738662691924b8a8f9b4b7cf1804923"
format_stamp: "Formatted at 2026-03-17 21:20:01 on dist-test-slave-979h"
I20260317 21:20:01.042133 9619 fs_manager.cc:696] Time spent creating directory manager: real 0.009s user 0.007s sys 0.001s
I20260317 21:20:01.049767 9650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:01.051309 9619 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260317 21:20:01.051728 9619 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal
uuid: "f738662691924b8a8f9b4b7cf1804923"
format_stamp: "Formatted at 2026-03-17 21:20:01 on dist-test-slave-979h"
I20260317 21:20:01.052086 9619 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:20:01.109655 9619 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:20:01.111455 9619 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:20:01.111929 9619 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:20:01.114881 9619 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:20:01.120261 9619 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:20:01.120514 9619 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:01.120913 9619 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:20:01.121258 9619 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:01.284250 9619 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.66:42819
I20260317 21:20:01.284380 9763 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.66:42819 every 8 connection(s)
I20260317 21:20:01.287539 9619 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/data/info.pb
I20260317 21:20:01.296401 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9619
I20260317 21:20:01.297258 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/instance
I20260317 21:20:01.325079 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.6.122.67:0
--local_ip_for_outbound_sockets=127.6.122.67
--webserver_interface=127.6.122.67
--webserver_port=0
--tserver_master_addrs=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20260317 21:20:01.347224 9766 heartbeater.cc:344] Connected to a master server at 127.6.122.126:32989
I20260317 21:20:01.348008 9766 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:01.349327 9766 heartbeater.cc:507] Master 127.6.122.126:32989 requested a full tablet report, sending...
I20260317 21:20:01.350939 9765 heartbeater.cc:344] Connected to a master server at 127.6.122.124:41015
I20260317 21:20:01.351045 9764 heartbeater.cc:344] Connected to a master server at 127.6.122.125:38261
I20260317 21:20:01.351354 9765 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:01.351439 9764 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:01.352312 9765 heartbeater.cc:507] Master 127.6.122.124:41015 requested a full tablet report, sending...
I20260317 21:20:01.352576 9764 heartbeater.cc:507] Master 127.6.122.125:38261 requested a full tablet report, sending...
I20260317 21:20:01.355697 9209 ts_manager.cc:194] Registered new tserver with Master: f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:01.361505 9363 ts_manager.cc:194] Registered new tserver with Master: f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:01.363428 9363 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.122.66:40355
I20260317 21:20:01.366441 9284 ts_manager.cc:194] Registered new tserver with Master: f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:01.638412 9775 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:01.638880 9775 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:01.640360 9775 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:01.641590 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:01.641568 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:20:01.643297 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:01.643991 9775 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:20:01.713624 9774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:20:01.714208 9774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:20:01.714844 9774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:20:01.751063 9774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:20:01.752487 9774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.6.122.67
I20260317 21:20:01.799449 9774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.6.122.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.6.122.67
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.6.122.67
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:20:01.800989 9774 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:20:01.802726 9774 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260317 21:20:01.816974 9782 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:01.817659 9782 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
W20260317 21:20:01.819206 9781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:01.821717 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:20:01.822880 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:01.824734 9782 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:01.824955 9268 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:01.826977 9782 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:01.872941 9765 heartbeater.cc:499] Master 127.6.122.124:41015 was elected leader, sending a full tablet report...
I20260317 21:20:02.275681 9788 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:02.279970 9788 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:02.286075 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:20:02.287194 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:02.291607 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:02.298938 9788 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:02.300601 9788 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:20:02.398869 9195 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.6.122.124:41015, user_credentials={real_user=slave}} blocked reactor thread for 107435us
I20260317 21:20:02.834470 9789 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:02.835285 9789 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:02.852223 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:02.856456 9789 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:02.859962 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:20:02.871771 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:02.905743 9789 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:03.339531 9791 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:03.340026 9791 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:03.345424 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:20:03.382896 9791 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:03.385587 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:03.390036 9270 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a4719f316c614a9cadaf5edab834804d; no voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:03.390870 9791 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:20:03.223439 9780 debug-util.cc:398] Leaking SignalData structure 0x7b0800002660 after lost signal to thread 9774
W20260317 21:20:03.443357 9780 kernel_stack_watchdog.cc:198] Thread 9774 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 393ms:
Kernel stack:
(could not read kernel stack)
User stack:
<Timed out: thread did not respond: maybe it is blocking signals>
W20260317 21:20:01.826102 9783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:03.449502 9792 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:03.450222 9792 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
W20260317 21:20:03.454381 9774 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.627s user 0.002s sys 0.007s
W20260317 21:20:03.454842 9774 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.628s user 0.003s sys 0.008s
W20260317 21:20:03.455245 9784 instance_detector.cc:116] could not retrieve GCE instance metadata: Timed out: curl timeout: Timeout was reached: Connection timed out after 1627 milliseconds
I20260317 21:20:03.456910 9792 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
W20260317 21:20:03.457795 9785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:03.457968 9373 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:03.458077 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
I20260317 21:20:03.458204 9774 server_base.cc:1055] Not found: could not retrieve instance metadata: unable to detect cloud type of this node, probably running in non-cloud environment
I20260317 21:20:03.460021 9774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:20:03.460613 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:03.461797 9792 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:03.463037 9774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:20:03.464578 9774 hybrid_clock.cc:648] HybridClock initialized: now 1773782403464502 us; error 81 us; skew 500 ppm
I20260317 21:20:03.465569 9774 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:20:03.473582 9774 webserver.cc:492] Webserver started at http://127.6.122.67:43587/ using document root <none> and password file <none>
I20260317 21:20:03.474627 9774 fs_manager.cc:362] Metadata directory not provided
I20260317 21:20:03.474920 9774 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:20:03.475456 9774 server_base.cc:908] This appears to be a new deployment of Kudu; creating new FS layout
I20260317 21:20:03.480453 9774 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data/instance:
uuid: "6c2dd7c5a524427da691785d88218218"
format_stamp: "Formatted at 2026-03-17 21:20:03 on dist-test-slave-979h"
I20260317 21:20:03.481849 9774 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/instance:
uuid: "6c2dd7c5a524427da691785d88218218"
format_stamp: "Formatted at 2026-03-17 21:20:03 on dist-test-slave-979h"
I20260317 21:20:03.490499 9774 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260317 21:20:03.498234 9797 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:03.499821 9774 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.004s
I20260317 21:20:03.500273 9774 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal
uuid: "6c2dd7c5a524427da691785d88218218"
format_stamp: "Formatted at 2026-03-17 21:20:03 on dist-test-slave-979h"
I20260317 21:20:03.500752 9774 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:20:03.563835 9774 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:20:03.565586 9774 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:20:03.566177 9774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:20:03.569308 9774 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260317 21:20:03.573985 9774 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260317 21:20:03.574219 9774 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:03.574570 9774 ts_tablet_manager.cc:616] Registered 0 tablets
I20260317 21:20:03.574765 9774 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:03.747385 9774 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.67:38595
I20260317 21:20:03.747504 9909 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.67:38595 every 8 connection(s)
I20260317 21:20:03.752795 9774 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/data/info.pb
I20260317 21:20:03.757274 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9774
I20260317 21:20:03.757912 6633 external_mini_cluster.cc:1442] Reading /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/instance
I20260317 21:20:03.806710 9911 heartbeater.cc:344] Connected to a master server at 127.6.122.124:41015
I20260317 21:20:03.807314 9911 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:03.808879 9910 heartbeater.cc:344] Connected to a master server at 127.6.122.125:38261
I20260317 21:20:03.808820 9911 heartbeater.cc:507] Master 127.6.122.124:41015 requested a full tablet report, sending...
I20260317 21:20:03.809288 9910 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:03.810155 9910 heartbeater.cc:507] Master 127.6.122.125:38261 requested a full tablet report, sending...
I20260317 21:20:03.810190 9912 heartbeater.cc:344] Connected to a master server at 127.6.122.126:32989
I20260317 21:20:03.810696 9912 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:03.811937 9912 heartbeater.cc:507] Master 127.6.122.126:32989 requested a full tablet report, sending...
I20260317 21:20:03.812150 9363 ts_manager.cc:194] Registered new tserver with Master: 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:03.814297 9363 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.6.122.67:36467
I20260317 21:20:03.816082 9284 ts_manager.cc:194] Registered new tserver with Master: 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:03.817174 9209 ts_manager.cc:194] Registered new tserver with Master: 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:03.823525 6633 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20260317 21:20:03.877103 9363 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51110:
name: "default.Table_26888b91e0784e639d5d19ef2ab9e027"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:03.885902 9361 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51110:
name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:03.888060 9360 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51110:
name: "default.Table_9d109e2778144950af7148124af0e79e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:03.889863 9362 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51110:
name: "default.Table_4bf9ab6d7efc481a9fc1306b6dbd2897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:20:03.909824 9363 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_26888b91e0784e639d5d19ef2ab9e027 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:20:03.914726 9362 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:20:03.918159 9361 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:20:03.920492 9360 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9d109e2778144950af7148124af0e79e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:20:03.939262 6633 external_mini_cluster.cc:1658] Killing /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu with pid 9320
W20260317 21:20:03.973419 9922 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.124:41015)
I20260317 21:20:03.977479 9944 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:03.979231 9944 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:03.991123 9944 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:03.992688 9293 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a4719f316c614a9cadaf5edab834804d" is_pre_election: true
W20260317 21:20:04.002324 9195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111) [suppressed 13 similar messages]
W20260317 21:20:04.021726 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015): Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:20:04.022369 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:04.023128 9944 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:04.264822 9948 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:04.266034 9948 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:04.269196 9948 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:04.270923 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac" is_pre_election: true
I20260317 21:20:04.271569 9217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a4719f316c614a9cadaf5edab834804d in term 1.
I20260317 21:20:04.277161 9268 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac; no voters:
W20260317 21:20:04.279465 9270 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111) [suppressed 6 similar messages]
I20260317 21:20:04.280277 9948 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260317 21:20:04.293110 9948 raft_consensus.cc:493] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Starting leader election (detected failure of leader ed19cc1cc9e74a0d9b8dc4a609f1bea9)
I20260317 21:20:04.293886 9948 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 1 FOLLOWER]: Advancing to term 2
W20260317 21:20:04.306228 9270 leader_election.cc:336] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015): Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:20:04.316304 9948 raft_consensus.cc:515] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:04.321437 9911 heartbeater.cc:499] Master 127.6.122.124:41015 was elected leader, sending a full tablet report...
I20260317 21:20:04.324854 9217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a4719f316c614a9cadaf5edab834804d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d6b9687775954282a11f2cf829d24bac"
I20260317 21:20:04.325501 9217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 1 FOLLOWER]: Advancing to term 2
W20260317 21:20:04.348053 9270 leader_election.cc:336] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015): Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:20:04.348616 9948 leader_election.cc:290] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 election: Requested vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:04.352286 9217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a4719f316c614a9cadaf5edab834804d in term 2.
I20260317 21:20:04.370134 9268 leader_election.cc:304] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac; no voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:04.372186 9948 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 2 FOLLOWER]: Leader election won for term 2
I20260317 21:20:04.372913 9948 raft_consensus.cc:697] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [term 2 LEADER]: Becoming Leader. State: Replica: a4719f316c614a9cadaf5edab834804d, State: Running, Role: LEADER
I20260317 21:20:04.374147 9948 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
W20260317 21:20:04.391438 9911 heartbeater.cc:646] Failed to heartbeat to 127.6.122.124:41015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
W20260317 21:20:04.393417 9616 heartbeater.cc:646] Failed to heartbeat to 127.6.122.124:41015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:20:04.405058 9950 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4719f316c614a9cadaf5edab834804d. Latest consensus state: current_term: 2 leader_uuid: "a4719f316c614a9cadaf5edab834804d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:04.405859 9950 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: This master's current role is: LEADER
I20260317 21:20:04.416239 9955 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:20:04.443986 9955 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:20:04.449524 9955 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:20:04.449920 9955 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:20:04.455288 9955 catalog_manager.cc:1514] Loading token signing keys...
W20260317 21:20:04.461697 9765 heartbeater.cc:646] Failed to heartbeat to 127.6.122.124:41015 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:20:04.484370 9955 catalog_manager.cc:6055] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Loaded TSK: 0
I20260317 21:20:04.485853 9955 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:20:04.489846 9955 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260317 21:20:04.494716 9955 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260317 21:20:04.517608 9283 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_9d109e2778144950af7148124af0e79e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:04.520195 9284 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:04.522127 9282 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_4bf9ab6d7efc481a9fc1306b6dbd2897"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:04.523917 9281 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_26888b91e0784e639d5d19ef2ab9e027"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260317 21:20:04.541841 9282 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:20:04.549541 9283 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_9d109e2778144950af7148124af0e79e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:20:04.550745 9284 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20260317 21:20:04.555173 9281 catalog_manager.cc:7033] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.table_26888b91e0784e639d5d19ef2ab9e027 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260317 21:20:04.767552 9281 catalog_manager.cc:2174] created HMS catalog entry for table [id=41bfb58e60114fa1a4b52e14d5ff15ed]
I20260317 21:20:04.784603 9217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Refusing update from remote peer a4719f316c614a9cadaf5edab834804d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260317 21:20:04.786675 9948 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260317 21:20:04.787992 9270 consensus_peers.cc:597] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d -> Peer ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015): Couldn't send request to peer ed19cc1cc9e74a0d9b8dc4a609f1bea9. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:20:04.828497 9958 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4719f316c614a9cadaf5edab834804d. Latest consensus state: current_term: 2 leader_uuid: "a4719f316c614a9cadaf5edab834804d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:04.834051 9958 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:04.839037 9282 catalog_manager.cc:2174] created HMS catalog entry for table [id=b14c029174b947aba867883bd0fd8019]
I20260317 21:20:04.841534 9957 mvcc.cc:204] Tried to move back new op lower bound from 7265412729978499072 to 7265412728360853504. Current Snapshot: MvccSnapshot[applied={T|T < 7265412729978499072}]
I20260317 21:20:04.846781 9959 mvcc.cc:204] Tried to move back new op lower bound from 7265412729978499072 to 7265412728360853504. Current Snapshot: MvccSnapshot[applied={T|T < 7265412729978499072}]
I20260317 21:20:04.847376 9948 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "a4719f316c614a9cadaf5edab834804d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:04.848172 9948 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d [sys.catalog]: This master's current role is: LEADER
I20260317 21:20:04.856773 9281 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35054:
table { table_name: "default.Table_26888b91e0784e639d5d19ef2ab9e027" } new_table_name: "default.Table_cc4ef5d4d3a54685ae63ef8a6b8f47de" modify_external_catalogs: true
I20260317 21:20:04.861857 9958 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "a4719f316c614a9cadaf5edab834804d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:04.862643 9958 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:04.919260 9282 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:35054:
table { table_name: "default.Table_4bf9ab6d7efc481a9fc1306b6dbd2897" } new_table_name: "default.Table_2e2f41675c9e4d4393bf153766901b13" modify_external_catalogs: true
I20260317 21:20:04.962010 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 84202923ed32477ebc0dfecd360ebdbc (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:04.963589 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:41015
--webserver_interface=127.6.122.124
--webserver_port=33267
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--hive_metastore_uris=thrift://127.0.0.1:42187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260317 21:20:04.964444 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84202923ed32477ebc0dfecd360ebdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:04.984200 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:04.985931 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3016575ec4c4489d9b5ba54f25d55a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:04.994048 9548 tablet_service.cc:1511] Processing CreateTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:04.996117 9548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3016575ec4c4489d9b5ba54f25d55a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:04.997087 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 84202923ed32477ebc0dfecd360ebdbc (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:04.999924 9697 tablet_service.cc:1511] Processing CreateTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.001279 9697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3016575ec4c4489d9b5ba54f25d55a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:04.967665 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 84202923ed32477ebc0dfecd360ebdbc (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.007421 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84202923ed32477ebc0dfecd360ebdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.008198 9284 catalog_manager.cc:2174] created HMS catalog entry for table [id=468218bf40694b02b7d92fc4da2d8962]
I20260317 21:20:05.006314 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84202923ed32477ebc0dfecd360ebdbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.046756 9284 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35054:
table { table_name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6" } modify_external_catalogs: true
I20260317 21:20:05.047874 9284 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35054:
table { table_name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6" } modify_external_catalogs: true
I20260317 21:20:05.058797 9973 tablet_bootstrap.cc:492] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:05.058707 9280 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_8dd898bb0e73488187cc030065d7ced6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:05.069366 9973 tablet_bootstrap.cc:654] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.072417 9973 log.cc:826] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Log is configured to *not* fsync() on all Append() calls
I20260317 21:20:05.082173 9279 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_c25a4ee82851411b8227cf1f4f04844e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:05.087756 9973 tablet_bootstrap.cc:492] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:05.088398 9973 ts_tablet_manager.cc:1403] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.030s user 0.023s sys 0.000s
I20260317 21:20:05.119042 9547 tablet_service.cc:1511] Processing CreateTablet for tablet 4154c79f1f8a4ff69165977da171e393 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.120496 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4154c79f1f8a4ff69165977da171e393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.121172 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 4154c79f1f8a4ff69165977da171e393 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.122326 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4154c79f1f8a4ff69165977da171e393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.123595 9843 tablet_service.cc:1511] Processing CreateTablet for tablet 4154c79f1f8a4ff69165977da171e393 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.125918 9843 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4154c79f1f8a4ff69165977da171e393. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.140861 9973 raft_consensus.cc:359] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.142025 9973 raft_consensus.cc:385] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.142421 9973 raft_consensus.cc:740] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.144447 9973 consensus_queue.cc:260] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.160665 9278 master_service.cc:954] ReplaceTablet: received request to replace tablet 3016575ec4c4489d9b5ba54f25d55a2b from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.161664 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 3016575ec4c4489d9b5ba54f25d55a2b from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.163389 9764 heartbeater.cc:499] Master 127.6.122.125:38261 was elected leader, sending a full tablet report...
I20260317 21:20:05.184898 9973 ts_tablet_manager.cc:1434] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.096s user 0.034s sys 0.008s
I20260317 21:20:05.187584 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 3016575ec4c4489d9b5ba54f25d55a2b
I20260317 21:20:05.189471 9973 tablet_bootstrap.cc:492] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:05.189684 9277 catalog_manager.cc:6741] ReplaceTablet: tablet 3016575ec4c4489d9b5ba54f25d55a2b deleted and replaced by tablet 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:05.192598 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b with delete_type TABLET_DATA_DELETED (replaced by tablet 24470256f195441ea55e67378bd4d5bf) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:05.203498 9973 tablet_bootstrap.cc:654] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.206480 9976 tablet_bootstrap.cc:492] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:05.208736 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b with delete_type TABLET_DATA_DELETED (replaced by tablet 24470256f195441ea55e67378bd4d5bf) from {username='slave'} at 127.0.0.1:33304
W20260317 21:20:05.215407 9270 catalog_manager.cc:4982] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): delete failed for tablet 3016575ec4c4489d9b5ba54f25d55a2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3016575ec4c4489d9b5ba54f25d55a2b already in progress: creating tablet
I20260317 21:20:05.220132 9973 tablet_bootstrap.cc:492] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:05.220818 9973 ts_tablet_manager.cc:1403] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.007s
I20260317 21:20:05.221860 9842 tablet_service.cc:1558] Processing DeleteTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b with delete_type TABLET_DATA_DELETED (replaced by tablet 24470256f195441ea55e67378bd4d5bf) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:05.224004 9973 raft_consensus.cc:359] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.225183 9976 tablet_bootstrap.cc:654] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.225255 9973 raft_consensus.cc:385] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.225666 9973 raft_consensus.cc:740] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.226573 9973 consensus_queue.cc:260] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.230623 9976 log.cc:826] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Log is configured to *not* fsync() on all Append() calls
I20260317 21:20:05.234162 9973 ts_tablet_manager.cc:1434] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.013s user 0.006s sys 0.000s
W20260317 21:20:05.238547 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 3016575ec4c4489d9b5ba54f25d55a2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3016575ec4c4489d9b5ba54f25d55a2b already in progress: creating tablet
I20260317 21:20:05.245970 9973 tablet_bootstrap.cc:492] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:05.246039 9276 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_9da3262f4c254404acf6cd7ddac1377c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:05.251025 9976 tablet_bootstrap.cc:492] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:05.251729 9976 ts_tablet_manager.cc:1403] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.005s
I20260317 21:20:05.256032 9973 tablet_bootstrap.cc:654] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.259070 9841 tablet_service.cc:1511] Processing CreateTablet for tablet 24470256f195441ea55e67378bd4d5bf (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.260599 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24470256f195441ea55e67378bd4d5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.261507 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 24470256f195441ea55e67378bd4d5bf (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.262776 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24470256f195441ea55e67378bd4d5bf. 1 dirs total, 0 dirs full, 0 dirs failed
W20260317 21:20:05.272441 9268 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 3016575ec4c4489d9b5ba54f25d55a2b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3016575ec4c4489d9b5ba54f25d55a2b already in progress: creating tablet
I20260317 21:20:05.275517 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 24470256f195441ea55e67378bd4d5bf (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.276852 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24470256f195441ea55e67378bd4d5bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.290052 9978 tablet_bootstrap.cc:492] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:05.306905 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b with delete_type TABLET_DATA_DELETED (replaced by tablet 24470256f195441ea55e67378bd4d5bf) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:05.307971 9979 tablet_replica.cc:333] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:05.308902 9979 raft_consensus.cc:2243] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.310348 9979 raft_consensus.cc:2272] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.310774 9978 tablet_bootstrap.cc:654] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.313581 9978 log.cc:826] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Log is configured to *not* fsync() on all Append() calls
I20260317 21:20:05.315841 9979 ts_tablet_manager.cc:1916] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:20:05.335870 9769 tablet.cc:2379] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:20:05.352669 9979 ts_tablet_manager.cc:1929] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.355116 9979 log.cc:1199] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/3016575ec4c4489d9b5ba54f25d55a2b
I20260317 21:20:05.357280 9979 ts_tablet_manager.cc:1950] T 3016575ec4c4489d9b5ba54f25d55a2b P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:05.361660 9976 raft_consensus.cc:359] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.362848 9976 raft_consensus.cc:385] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.363202 9976 raft_consensus.cc:740] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.364622 9976 consensus_queue.cc:260] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.378738 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 3016575ec4c4489d9b5ba54f25d55a2b (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:05.380036 9978 tablet_bootstrap.cc:492] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:05.380765 9978 ts_tablet_manager.cc:1403] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.092s user 0.032s sys 0.033s
I20260317 21:20:05.381348 9973 tablet_bootstrap.cc:492] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:05.381882 9973 ts_tablet_manager.cc:1403] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.136s user 0.027s sys 0.016s
I20260317 21:20:05.384961 9973 raft_consensus.cc:359] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.385821 9973 raft_consensus.cc:385] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.386076 9615 heartbeater.cc:499] Master 127.6.122.125:38261 was elected leader, sending a full tablet report...
I20260317 21:20:05.386559 9973 raft_consensus.cc:740] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.387427 9973 consensus_queue.cc:260] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.389300 9976 ts_tablet_manager.cc:1434] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.137s user 0.043s sys 0.000s
I20260317 21:20:05.390614 9976 tablet_bootstrap.cc:492] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:05.403972 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 24470256f195441ea55e67378bd4d5bf from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.406503 9275 master_service.cc:954] ReplaceTablet: received request to replace tablet 4154c79f1f8a4ff69165977da171e393 from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.408895 9976 tablet_bootstrap.cc:654] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.426594 9910 heartbeater.cc:499] Master 127.6.122.125:38261 was elected leader, sending a full tablet report...
I20260317 21:20:05.469887 9978 raft_consensus.cc:359] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.471122 9978 raft_consensus.cc:385] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.471836 9978 raft_consensus.cc:740] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.472043 9275 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 4154c79f1f8a4ff69165977da171e393
I20260317 21:20:05.474040 9275 catalog_manager.cc:6741] ReplaceTablet: tablet 4154c79f1f8a4ff69165977da171e393 deleted and replaced by tablet d1518f4f44504899a80615d9694c7e21
I20260317 21:20:05.478024 9977 raft_consensus.cc:493] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:05.478659 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:05.478958 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:05.479166 9977 raft_consensus.cc:515] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.479988 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:05.486184 9977 leader_election.cc:290] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:05.488289 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 4154c79f1f8a4ff69165977da171e393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4154c79f1f8a4ff69165977da171e393 already in progress: creating tablet
I20260317 21:20:05.493559 9973 ts_tablet_manager.cc:1434] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.111s user 0.003s sys 0.004s
I20260317 21:20:05.494980 9973 tablet_bootstrap.cc:492] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
W20260317 21:20:05.495872 9623 tablet.cc:2379] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260317 21:20:05.500991 9268 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 4154c79f1f8a4ff69165977da171e393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4154c79f1f8a4ff69165977da171e393 already in progress: creating tablet
I20260317 21:20:05.502125 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:05.503685 9277 catalog_manager.cc:6741] ReplaceTablet: tablet 24470256f195441ea55e67378bd4d5bf deleted and replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2
I20260317 21:20:05.505183 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:34012
W20260317 21:20:05.508302 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 24470256f195441ea55e67378bd4d5bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24470256f195441ea55e67378bd4d5bf already in progress: creating tablet
I20260317 21:20:05.508392 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:33304
W20260317 21:20:05.512501 9268 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 24470256f195441ea55e67378bd4d5bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24470256f195441ea55e67378bd4d5bf already in progress: creating tablet
I20260317 21:20:05.473605 9978 consensus_queue.cc:260] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.518239 9976 tablet_bootstrap.cc:492] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:05.522356 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b with delete_type TABLET_DATA_DELETED (replaced by tablet 24470256f195441ea55e67378bd4d5bf) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:05.523748 9982 tablet_replica.cc:333] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:05.523957 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:05.525183 9982 raft_consensus.cc:2243] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.525542 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:05.525964 9982 raft_consensus.cc:2272] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.529456 9982 ts_tablet_manager.cc:1916] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:05.533145 9976 ts_tablet_manager.cc:1403] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.143s user 0.016s sys 0.016s
W20260317 21:20:05.534514 9270 catalog_manager.cc:4982] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): delete failed for tablet 4154c79f1f8a4ff69165977da171e393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4154c79f1f8a4ff69165977da171e393 already in progress: creating tablet
I20260317 21:20:05.539672 9976 raft_consensus.cc:359] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.541049 9973 tablet_bootstrap.cc:654] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.545820 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:47636
W20260317 21:20:05.547844 9270 catalog_manager.cc:4982] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): delete failed for tablet 24470256f195441ea55e67378bd4d5bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24470256f195441ea55e67378bd4d5bf already in progress: creating tablet
I20260317 21:20:05.555308 9978 ts_tablet_manager.cc:1434] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.174s user 0.036s sys 0.007s
I20260317 21:20:05.561061 9976 raft_consensus.cc:385] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.562979 9976 raft_consensus.cc:740] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.571445 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:05.572430 9979 tablet_replica.cc:333] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:05.573590 9979 raft_consensus.cc:2243] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.574362 9979 raft_consensus.cc:2272] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.575078 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:05.575078 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 3016575ec4c4489d9b5ba54f25d55a2b with delete_type TABLET_DATA_DELETED (replaced by tablet 24470256f195441ea55e67378bd4d5bf) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:05.576948 9981 tablet_replica.cc:333] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:05.578181 9979 ts_tablet_manager.cc:1916] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:05.578758 9981 raft_consensus.cc:2243] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.579432 9981 raft_consensus.cc:2272] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.582101 9698 tablet_service.cc:1511] Processing CreateTablet for tablet d1518f4f44504899a80615d9694c7e21 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.583395 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1518f4f44504899a80615d9694c7e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.585218 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:05.585757 9844 tablet_service.cc:1511] Processing CreateTablet for tablet d1518f4f44504899a80615d9694c7e21 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:20:05.586099 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 24470256f195441ea55e67378bd4d5bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24470256f195441ea55e67378bd4d5bf already in progress: creating tablet
I20260317 21:20:05.587044 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1518f4f44504899a80615d9694c7e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.587347 9973 tablet_bootstrap.cc:492] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:05.586961 9981 ts_tablet_manager.cc:1916] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:05.587839 9973 ts_tablet_manager.cc:1403] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.093s user 0.013s sys 0.000s
I20260317 21:20:05.596786 9549 tablet_service.cc:1511] Processing CreateTablet for tablet d1518f4f44504899a80615d9694c7e21 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.597982 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1518f4f44504899a80615d9694c7e21. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.601609 9841 tablet_service.cc:1511] Processing CreateTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.602818 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f2f33fb3744ae0b8db16d529a9def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.602428 9976 consensus_queue.cc:260] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.613173 9697 tablet_service.cc:1511] Processing CreateTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.614426 9697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f2f33fb3744ae0b8db16d529a9def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.614298 9547 tablet_service.cc:1511] Processing CreateTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.615542 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7f2f33fb3744ae0b8db16d529a9def2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.621438 9978 tablet_bootstrap.cc:492] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:05.607087 9973 raft_consensus.cc:359] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:05.628968 9973 raft_consensus.cc:385] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.629325 9973 raft_consensus.cc:740] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.630578 9976 ts_tablet_manager.cc:1434] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.094s user 0.007s sys 0.000s
I20260317 21:20:05.637145 9973 consensus_queue.cc:260] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:05.639962 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84202923ed32477ebc0dfecd360ebdbc" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:05.640991 9569 raft_consensus.cc:2468] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 0.
I20260317 21:20:05.643636 9651 leader_election.cc:304] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:05.645398 9977 raft_consensus.cc:2804] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:05.645828 9977 raft_consensus.cc:493] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:05.646158 9977 raft_consensus.cc:3060] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:05.642997 9973 ts_tablet_manager.cc:1434] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.055s user 0.003s sys 0.007s
I20260317 21:20:05.657773 9976 tablet_bootstrap.cc:492] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:05.675807 9973 tablet_bootstrap.cc:492] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:05.676335 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84202923ed32477ebc0dfecd360ebdbc" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
W20260317 21:20:05.679527 9652 leader_election.cc:343] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:05.684561 9977 raft_consensus.cc:515] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.687536 9275 master_service.cc:954] ReplaceTablet: received request to replace tablet 24470256f195441ea55e67378bd4d5bf from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.687603 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84202923ed32477ebc0dfecd360ebdbc" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4"
I20260317 21:20:05.688254 9569 raft_consensus.cc:3060] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:05.694737 9978 tablet_bootstrap.cc:654] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.702826 9976 tablet_bootstrap.cc:654] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.702826 9973 tablet_bootstrap.cc:654] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.707902 9569 raft_consensus.cc:2468] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 1.
I20260317 21:20:05.708863 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:05.709856 9981 ts_tablet_manager.cc:1929] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.710335 9981 log.cc:1199] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/3016575ec4c4489d9b5ba54f25d55a2b
I20260317 21:20:05.711455 9981 ts_tablet_manager.cc:1950] T 3016575ec4c4489d9b5ba54f25d55a2b P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:05.720191 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 84202923ed32477ebc0dfecd360ebdbc from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.721274 9982 ts_tablet_manager.cc:1929] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.722043 9982 log.cc:1199] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/3016575ec4c4489d9b5ba54f25d55a2b
I20260317 21:20:05.725232 9982 ts_tablet_manager.cc:1950] T 3016575ec4c4489d9b5ba54f25d55a2b P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:05.727600 9979 ts_tablet_manager.cc:1929] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.731469 9979 log.cc:1199] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/4154c79f1f8a4ff69165977da171e393
I20260317 21:20:05.733103 9979 ts_tablet_manager.cc:1950] T 4154c79f1f8a4ff69165977da171e393 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:05.734119 9978 tablet_bootstrap.cc:492] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:05.734620 9978 ts_tablet_manager.cc:1403] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.113s user 0.009s sys 0.007s
I20260317 21:20:05.740044 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 4154c79f1f8a4ff69165977da171e393 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:05.741487 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 3016575ec4c4489d9b5ba54f25d55a2b (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
W20260317 21:20:05.743788 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 4154c79f1f8a4ff69165977da171e393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4154c79f1f8a4ff69165977da171e393 already in progress: creating tablet
I20260317 21:20:05.744977 9978 raft_consensus.cc:359] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.745749 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 84202923ed32477ebc0dfecd360ebdbc
I20260317 21:20:05.746371 9979 tablet_replica.cc:333] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:05.747167 9979 raft_consensus.cc:2243] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.747314 9277 catalog_manager.cc:6741] ReplaceTablet: tablet 84202923ed32477ebc0dfecd360ebdbc deleted and replaced by tablet 08149c2a49274388be9400bdc390fab1
I20260317 21:20:05.747725 9979 raft_consensus.cc:2272] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.752008 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 84202923ed32477ebc0dfecd360ebdbc with delete_type TABLET_DATA_DELETED (replaced by tablet 08149c2a49274388be9400bdc390fab1) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:05.755297 9651 leader_election.cc:304] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:05.754570 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 84202923ed32477ebc0dfecd360ebdbc with delete_type TABLET_DATA_DELETED (replaced by tablet 08149c2a49274388be9400bdc390fab1) from {username='slave'} at 127.0.0.1:34012
W20260317 21:20:05.762077 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 84202923ed32477ebc0dfecd360ebdbc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 84202923ed32477ebc0dfecd360ebdbc already in progress: creating tablet
I20260317 21:20:05.762974 9978 raft_consensus.cc:385] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.764420 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 3016575ec4c4489d9b5ba54f25d55a2b (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
W20260317 21:20:05.765232 9268 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 24470256f195441ea55e67378bd4d5bf because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24470256f195441ea55e67378bd4d5bf already in progress: creating tablet
I20260317 21:20:05.765507 9976 tablet_bootstrap.cc:492] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:05.765722 9978 raft_consensus.cc:740] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.766041 9976 ts_tablet_manager.cc:1403] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.108s user 0.014s sys 0.003s
W20260317 21:20:05.765967 9268 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 4154c79f1f8a4ff69165977da171e393 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4154c79f1f8a4ff69165977da171e393 already in progress: creating tablet
I20260317 21:20:05.767624 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 84202923ed32477ebc0dfecd360ebdbc with delete_type TABLET_DATA_DELETED (replaced by tablet 08149c2a49274388be9400bdc390fab1) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:05.768051 9977 leader_election.cc:290] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:05.768743 9977 raft_consensus.cc:2804] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:20:05.768530 9982 tablet_replica.cc:333] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:05.769829 9977 raft_consensus.cc:697] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 1 LEADER]: Becoming Leader. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Running, Role: LEADER
I20260317 21:20:05.770104 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "84202923ed32477ebc0dfecd360ebdbc" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218"
I20260317 21:20:05.770290 9982 raft_consensus.cc:2243] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.770782 9982 raft_consensus.cc:2272] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.770860 9977 consensus_queue.cc:237] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.773347 9982 ts_tablet_manager.cc:1916] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:20:05.772904 9652 leader_election.cc:343] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:05.778414 9976 raft_consensus.cc:359] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.779291 9976 raft_consensus.cc:385] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.779619 9976 raft_consensus.cc:740] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.780942 9979 ts_tablet_manager.cc:1916] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:05.782749 9973 tablet_bootstrap.cc:492] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:05.783218 9973 ts_tablet_manager.cc:1403] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.008s
I20260317 21:20:05.790326 9978 consensus_queue.cc:260] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.780323 9976 consensus_queue.cc:260] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.802783 9976 ts_tablet_manager.cc:1434] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.036s user 0.002s sys 0.004s
I20260317 21:20:05.804196 9976 tablet_bootstrap.cc:492] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:05.807585 9978 ts_tablet_manager.cc:1434] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.073s user 0.012s sys 0.000s
I20260317 21:20:05.821172 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:05.822029 9981 tablet_replica.cc:333] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:05.822767 9981 raft_consensus.cc:2243] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:05.823300 9981 raft_consensus.cc:2272] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.796489 9973 raft_consensus.cc:359] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.825513 9973 raft_consensus.cc:385] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:05.825736 9981 ts_tablet_manager.cc:1916] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:05.825963 9973 raft_consensus.cc:740] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:05.826858 9973 consensus_queue.cc:260] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:05.846410 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 4154c79f1f8a4ff69165977da171e393 with delete_type TABLET_DATA_DELETED (replaced by tablet d1518f4f44504899a80615d9694c7e21) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:05.848793 9973 ts_tablet_manager.cc:1434] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.065s user 0.010s sys 0.000s
I20260317 21:20:05.851819 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 08149c2a49274388be9400bdc390fab1 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.853610 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08149c2a49274388be9400bdc390fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.857255 9981 ts_tablet_manager.cc:1929] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.858194 9981 log.cc:1199] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/4154c79f1f8a4ff69165977da171e393
I20260317 21:20:05.859319 9981 ts_tablet_manager.cc:1950] T 4154c79f1f8a4ff69165977da171e393 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:05.859913 9978 tablet_bootstrap.cc:492] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:05.862679 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 4154c79f1f8a4ff69165977da171e393 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:05.865172 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 08149c2a49274388be9400bdc390fab1 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.866036 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 08149c2a49274388be9400bdc390fab1 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:05.866515 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08149c2a49274388be9400bdc390fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.867033 9973 tablet_bootstrap.cc:492] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:05.867239 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 08149c2a49274388be9400bdc390fab1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:05.907331 9979 ts_tablet_manager.cc:1929] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.917383 9979 log.cc:1199] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/24470256f195441ea55e67378bd4d5bf
I20260317 21:20:05.924548 9979 ts_tablet_manager.cc:1950] T 24470256f195441ea55e67378bd4d5bf P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:05.925808 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 24470256f195441ea55e67378bd4d5bf from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.926718 9976 tablet_bootstrap.cc:654] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.929072 9979 tablet_replica.cc:333] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:05.929563 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 24470256f195441ea55e67378bd4d5bf (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:05.936936 9978 tablet_bootstrap.cc:654] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.938086 9979 raft_consensus.cc:2243] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:20:05.939253 9979 pending_rounds.cc:70] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Trying to abort 1 pending ops.
I20260317 21:20:05.940193 9979 pending_rounds.cc:77] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7265412734105526272 op_type: NO_OP noop_request { }
I20260317 21:20:05.948897 9979 raft_consensus.cc:2889] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260317 21:20:05.954813 9979 raft_consensus.cc:2272] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:05.957746 9979 ts_tablet_manager.cc:1916] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:05.962806 9275 master_service.cc:954] ReplaceTablet: received request to replace tablet 84202923ed32477ebc0dfecd360ebdbc from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:05.951473 9973 tablet_bootstrap.cc:654] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:05.968596 9982 ts_tablet_manager.cc:1929] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:05.991545 9976 tablet_bootstrap.cc:492] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:05.992058 9976 ts_tablet_manager.cc:1403] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.188s user 0.014s sys 0.000s
I20260317 21:20:05.995738 9982 log.cc:1199] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/84202923ed32477ebc0dfecd360ebdbc
I20260317 21:20:05.996841 9982 ts_tablet_manager.cc:1950] T 84202923ed32477ebc0dfecd360ebdbc P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:05.997014 9979 ts_tablet_manager.cc:1929] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:05.997797 9979 log.cc:1199] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/84202923ed32477ebc0dfecd360ebdbc
I20260317 21:20:05.999572 9979 ts_tablet_manager.cc:1950] T 84202923ed32477ebc0dfecd360ebdbc P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:05.999951 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 84202923ed32477ebc0dfecd360ebdbc (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.002671 9978 tablet_bootstrap.cc:492] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.002669 9973 tablet_bootstrap.cc:492] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:06.003175 9973 ts_tablet_manager.cc:1403] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.136s user 0.013s sys 0.003s
I20260317 21:20:06.005018 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 84202923ed32477ebc0dfecd360ebdbc (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.006218 9978 ts_tablet_manager.cc:1403] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.147s user 0.009s sys 0.005s
I20260317 21:20:06.007315 9982 tablet_replica.cc:333] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.008045 9982 raft_consensus.cc:2243] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.008596 9982 raft_consensus.cc:2272] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.007874 9976 raft_consensus.cc:359] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.010455 9976 raft_consensus.cc:385] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.010797 9976 raft_consensus.cc:740] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.012210 9978 raft_consensus.cc:359] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.013268 9978 raft_consensus.cc:385] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.013608 9978 raft_consensus.cc:740] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.015617 9978 consensus_queue.cc:260] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.018702 9982 ts_tablet_manager.cc:1916] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.020505 9976 consensus_queue.cc:260] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.007143 9973 raft_consensus.cc:359] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.023335 9973 raft_consensus.cc:385] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.023686 9973 raft_consensus.cc:740] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.028026 9976 ts_tablet_manager.cc:1434] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.036s user 0.008s sys 0.001s
I20260317 21:20:06.027534 9973 consensus_queue.cc:260] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.030103 9973 ts_tablet_manager.cc:1434] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.027s user 0.004s sys 0.005s
I20260317 21:20:06.034327 9978 ts_tablet_manager.cc:1434] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.028s user 0.006s sys 0.001s
I20260317 21:20:06.036866 9973 tablet_bootstrap.cc:492] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:06.040162 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 24470256f195441ea55e67378bd4d5bf with delete_type TABLET_DATA_DELETED (replaced by tablet d7f2f33fb3744ae0b8db16d529a9def2) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.043058 9973 tablet_bootstrap.cc:654] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.052966 9978 tablet_bootstrap.cc:492] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
W20260317 21:20:06.057265 9915 tablet.cc:2379] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260317 21:20:06.068729 9978 tablet_bootstrap.cc:654] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.071995 9976 tablet_bootstrap.cc:492] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.075356 9973 tablet_bootstrap.cc:492] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:06.075802 9973 ts_tablet_manager.cc:1403] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.039s user 0.012s sys 0.000s
I20260317 21:20:06.078092 9973 raft_consensus.cc:359] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.078779 9973 raft_consensus.cc:385] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.079030 9973 raft_consensus.cc:740] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.079636 9973 consensus_queue.cc:260] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.085460 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 84202923ed32477ebc0dfecd360ebdbc with delete_type TABLET_DATA_DELETED (replaced by tablet 08149c2a49274388be9400bdc390fab1) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.086449 9981 tablet_replica.cc:333] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:06.087229 9981 raft_consensus.cc:2243] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.087720 9981 raft_consensus.cc:2272] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:20:06.088331 9995 raft_consensus.cc:670] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: failed to trigger leader election: Illegal state: RaftConsensus is not running: State = Stopped
I20260317 21:20:06.088606 9973 ts_tablet_manager.cc:1434] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.013s user 0.006s sys 0.001s
I20260317 21:20:06.103915 9981 ts_tablet_manager.cc:1916] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:20:06.105602 9277 catalog_manager.cc:5504] Ignoring report from unknown tablet 84202923ed32477ebc0dfecd360ebdbc
I20260317 21:20:06.110988 9976 tablet_bootstrap.cc:654] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.114957 9982 ts_tablet_manager.cc:1929] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.115439 9982 log.cc:1199] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/4154c79f1f8a4ff69165977da171e393
I20260317 21:20:06.116137 9982 ts_tablet_manager.cc:1950] T 4154c79f1f8a4ff69165977da171e393 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.126621 9978 tablet_bootstrap.cc:492] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.127620 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 4154c79f1f8a4ff69165977da171e393 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:06.128611 9978 ts_tablet_manager.cc:1403] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.074s user 0.011s sys 0.005s
I20260317 21:20:06.129115 9982 tablet_replica.cc:333] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.129905 9982 raft_consensus.cc:2243] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.131052 9982 raft_consensus.cc:2272] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.131999 9978 raft_consensus.cc:359] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.133103 9978 raft_consensus.cc:385] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.133589 9978 raft_consensus.cc:740] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.134554 9976 tablet_bootstrap.cc:492] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:06.134298 9978 consensus_queue.cc:260] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.135017 9976 ts_tablet_manager.cc:1403] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.063s user 0.009s sys 0.004s
I20260317 21:20:06.136780 9978 ts_tablet_manager.cc:1434] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.008s user 0.002s sys 0.005s
I20260317 21:20:06.140455 9976 raft_consensus.cc:359] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.141281 9976 raft_consensus.cc:385] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.141593 9976 raft_consensus.cc:740] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.143826 9278 master_service.cc:954] ReplaceTablet: received request to replace tablet 24470256f195441ea55e67378bd4d5bf from {username='slave'} at 127.0.0.1:35054
W20260317 21:20:06.147939 9278 catalog_manager.cc:5504] Ignoring report from unknown tablet 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:06.149755 9982 ts_tablet_manager.cc:1916] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.158529 9978 tablet_bootstrap.cc:492] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
W20260317 21:20:06.160235 9278 catalog_manager.cc:5504] Ignoring report from unknown tablet 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:06.172763 9278 master_service.cc:954] ReplaceTablet: received request to replace tablet d7f2f33fb3744ae0b8db16d529a9def2 from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:06.146957 9976 consensus_queue.cc:260] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.192420 9976 ts_tablet_manager.cc:1434] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.057s user 0.008s sys 0.000s
I20260317 21:20:06.193528 9976 tablet_bootstrap.cc:492] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.196048 9977 raft_consensus.cc:493] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:06.196703 9977 raft_consensus.cc:515] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.198487 9278 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet d7f2f33fb3744ae0b8db16d529a9def2
I20260317 21:20:06.200058 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f2f33fb3744ae0b8db16d529a9def2" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
I20260317 21:20:06.200876 9278 catalog_manager.cc:6741] ReplaceTablet: tablet d7f2f33fb3744ae0b8db16d529a9def2 deleted and replaced by tablet b8519cc49ca34a5fa0941386938ea7fc
I20260317 21:20:06.201341 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 with delete_type TABLET_DATA_DELETED (replaced by tablet b8519cc49ca34a5fa0941386938ea7fc) from {username='slave'} at 127.0.0.1:33304
W20260317 21:20:06.201491 9652 leader_election.cc:343] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:06.203972 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 with delete_type TABLET_DATA_DELETED (replaced by tablet b8519cc49ca34a5fa0941386938ea7fc) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.206329 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 with delete_type TABLET_DATA_DELETED (replaced by tablet b8519cc49ca34a5fa0941386938ea7fc) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:06.207185 9979 tablet_replica.cc:333] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:06.207453 9978 tablet_bootstrap.cc:654] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.208005 9979 raft_consensus.cc:2243] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.206391 9977 leader_election.cc:290] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:06.208523 9979 raft_consensus.cc:2272] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.212096 9981 ts_tablet_manager.cc:1929] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.212550 9981 log.cc:1199] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/84202923ed32477ebc0dfecd360ebdbc
I20260317 21:20:06.213343 9981 ts_tablet_manager.cc:1950] T 84202923ed32477ebc0dfecd360ebdbc P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:06.216203 9982 ts_tablet_manager.cc:1929] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.217188 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 84202923ed32477ebc0dfecd360ebdbc (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.221676 9976 tablet_bootstrap.cc:654] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.227509 9982 log.cc:1199] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/24470256f195441ea55e67378bd4d5bf
I20260317 21:20:06.228325 9982 ts_tablet_manager.cc:1950] T 24470256f195441ea55e67378bd4d5bf P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.223948 9979 ts_tablet_manager.cc:1916] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.231230 9982 tablet_replica.cc:333] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.231935 9982 raft_consensus.cc:2243] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.232410 9982 raft_consensus.cc:2272] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
21:20:06.236 [HMS - ERROR - pool-6-thread-2] (RetryingHMSHandler.java:201) AlreadyExistsException(message:Table hive.default.table_9d109e2778144950af7148124af0e79e already exists)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1839)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_core(HiveMetaStore.java:1786)
at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.create_table_with_environment_context(HiveMetaStore.java:2035)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:569)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invokeInternal(RetryingHMSHandler.java:147)
at org.apache.hadoop.hive.metastore.RetryingHMSHandler.invoke(RetryingHMSHandler.java:108)
at jdk.proxy2/jdk.proxy2.$Proxy27.create_table_with_environment_context(Unknown Source)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14800)
at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Processor$create_table_with_environment_context.getResult(ThriftHiveMetastore.java:14784)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:39)
at org.apache.hadoop.hive.metastore.TUGIBasedProcessor.process(TUGIBasedProcessor.java:104)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:286)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:840)
I20260317 21:20:06.238451 9982 ts_tablet_manager.cc:1916] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260317 21:20:06.241353 9436 hms_client.cc:285] Time spent create HMS table: real 1.234s user 0.000s sys 0.002s
W20260317 21:20:06.243614 9283 catalog_manager.cc:2170] Already present: failed to create HMS catalog entry for table [id=6a9a9c7a2a79412b929ac4aca31dd922]: failed to create Hive MetaStore table: TException - service has thrown: AlreadyExistsException(message=Table hive.default.table_9d109e2778144950af7148124af0e79e already exists)
I20260317 21:20:06.244546 9283 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:35054 (request call id 22) took 1729 ms. Trace:
I20260317 21:20:06.244944 9283 rpcz_store.cc:276] 0317 21:20:04.515606 (+ 0us) service_pool.cc:168] Inserting onto call queue
0317 21:20:04.516895 (+ 1289us) service_pool.cc:225] Handling call
0317 21:20:04.550035 (+ 33140us) catalog_manager.cc:2101] Acquired catalog manager lock
0317 21:20:04.560935 (+ 10900us) catalog_manager.cc:2147] Created new table and tablet info
0317 21:20:06.244474 (+1683539us) inbound_call.cc:173] Queueing success response
Metrics: {"Hive Metastore.queue_time_us":443215,"Hive Metastore.run_cpu_time_us":2593,"Hive Metastore.run_wall_time_us":1234357,"spinlock_wait_cycles":6016}
I20260317 21:20:06.248796 9977 raft_consensus.cc:493] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260317 21:20:06.249311 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet d7f2f33fb3744ae0b8db16d529a9def2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7f2f33fb3744ae0b8db16d529a9def2 already in progress: creating tablet
I20260317 21:20:06.249295 9977 raft_consensus.cc:515] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.251271 9977 leader_election.cc:290] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:06.255008 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1518f4f44504899a80615d9694c7e21" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
W20260317 21:20:06.256862 9652 leader_election.cc:343] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:06.258982 9283 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35054:
name: "default.Table_66e3c05d553648b4aab0c59288b3caee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:06.280619 9698 tablet_service.cc:1511] Processing CreateTablet for tablet b8519cc49ca34a5fa0941386938ea7fc (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.281926 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8519cc49ca34a5fa0941386938ea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.293541 9844 tablet_service.cc:1511] Processing CreateTablet for tablet b8519cc49ca34a5fa0941386938ea7fc (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.294759 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8519cc49ca34a5fa0941386938ea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
F20260317 21:20:06.299096 9439 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1773782404, eventType=CREATE_TABLE, dbName=default, tableName=table_26888b91e0784e639d5d19ef2ab9e027, message={"server":"","servicePrincipal":"","db":"default","table":"table_26888b91e0784e639d5d19ef2ab9e027","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_26888b91e0784e639d5d19ef2ab9e027\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1773782404},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/warehouse/table_26888b91e0784e639d5d19ef2ab9e027\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_26888b91e0784e639d5d19ef2ab9e027\",\"kudu.table_id\":\"41bfb58e60114fa1a4b52e14d5ff15ed\",\"transient_lastDdlTime\":\"1773782404\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"55c0c553ed67449ba5bba0d7fbc16229\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1773782404,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1773782404, eventType=CREATE_TABLE, dbName=default, tableName=table_9d109e2778144950af7148124af0e79e, message={"server":"","servicePrincipal":"","db":"default","table":"table_9d109e2778144950af7148124af0e79e","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9d109e2778144950af7148124af0e79e\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1773782404},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/warehouse/table_9d109e2778144950af7148124af0e79e\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9d109e2778144950af7148124af0e79e\",\"kudu.table_id\":\"1dc9b1c3c6e847d08861a08accef3b80\",\"transient_lastDdlTime\":\"1773782404\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"55c0c553ed67449ba5bba0d7fbc16229\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1773782404,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260317 21:20:06.322849 9978 tablet_bootstrap.cc:492] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.323412 9978 ts_tablet_manager.cc:1403] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.165s user 0.022s sys 0.002s
I20260317 21:20:06.326395 9978 raft_consensus.cc:359] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.327229 9978 raft_consensus.cc:385] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.327572 9978 raft_consensus.cc:740] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.328418 9978 consensus_queue.cc:260] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.333096 9978 ts_tablet_manager.cc:1434] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.009s user 0.006s sys 0.000s
I20260317 21:20:06.334179 9978 tablet_bootstrap.cc:492] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:06.340413 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 24470256f195441ea55e67378bd4d5bf (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:06.341764 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 with delete_type TABLET_DATA_DELETED (replaced by tablet b8519cc49ca34a5fa0941386938ea7fc) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.342589 9981 tablet_replica.cc:333] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:06.343408 9981 raft_consensus.cc:2243] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.343943 9981 raft_consensus.cc:2272] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.347199 9981 ts_tablet_manager.cc:1916] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.357919 9278 master_service.cc:954] ReplaceTablet: received request to replace tablet 08149c2a49274388be9400bdc390fab1 from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:06.360075 9549 tablet_service.cc:1511] Processing CreateTablet for tablet b8519cc49ca34a5fa0941386938ea7fc (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.361835 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8519cc49ca34a5fa0941386938ea7fc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd4ee350950 google::LogMessage::Flush() at ??:0
I20260317 21:20:06.364514 9978 tablet_bootstrap.cc:654] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.373811 9568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d1518f4f44504899a80615d9694c7e21" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:06.276521 9982 ts_tablet_manager.cc:1929] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.376597 9982 log.cc:1199] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/d7f2f33fb3744ae0b8db16d529a9def2
I20260317 21:20:06.377911 9982 ts_tablet_manager.cc:1950] T d7f2f33fb3744ae0b8db16d529a9def2 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.373808 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d7f2f33fb3744ae0b8db16d529a9def2" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:06.381063 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet d7f2f33fb3744ae0b8db16d529a9def2 (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:06.383546 9976 tablet_bootstrap.cc:492] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
W20260317 21:20:06.383841 9651 leader_election.cc:343] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:06.384076 9976 ts_tablet_manager.cc:1403] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.191s user 0.018s sys 0.007s
I20260317 21:20:06.384320 9651 leader_election.cc:304] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:06.387262 9976 raft_consensus.cc:359] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.388139 9976 raft_consensus.cc:385] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.388458 9976 raft_consensus.cc:740] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.388669 9981 ts_tablet_manager.cc:1929] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.389070 9981 log.cc:1199] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/d7f2f33fb3744ae0b8db16d529a9def2
I20260317 21:20:06.388962 9977 raft_consensus.cc:2749] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:06.389771 9981 ts_tablet_manager.cc:1950] T d7f2f33fb3744ae0b8db16d529a9def2 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:06.389259 9976 consensus_queue.cc:260] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.391603 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet d7f2f33fb3744ae0b8db16d529a9def2 with delete_type TABLET_DATA_DELETED (replaced by tablet b8519cc49ca34a5fa0941386938ea7fc) from {username='slave'} at 127.0.0.1:33304
W20260317 21:20:06.392165 9651 leader_election.cc:343] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Illegal state: Tablet not RUNNING: STOPPED
I20260317 21:20:06.392627 9651 leader_election.cc:304] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:06.392915 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet d7f2f33fb3744ae0b8db16d529a9def2 (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
W20260317 21:20:06.393265 9651 raft_consensus.cc:2686] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: unable to run election callback: Service unavailable: Thread pool token was shut down
W20260317 21:20:06.393463 9268 catalog_manager.cc:4969] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet d7f2f33fb3744ae0b8db16d529a9def2 because the tablet was not found. No further retry: Not found: Tablet not found: d7f2f33fb3744ae0b8db16d529a9def2
I20260317 21:20:06.395692 9978 tablet_bootstrap.cc:492] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.396153 9978 ts_tablet_manager.cc:1403] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.062s user 0.011s sys 0.000s
I20260317 21:20:06.398835 9976 ts_tablet_manager.cc:1434] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.014s user 0.001s sys 0.008s
I20260317 21:20:06.400137 9976 tablet_bootstrap.cc:492] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.404899 9978 raft_consensus.cc:359] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.405714 9978 raft_consensus.cc:385] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.406194 9978 raft_consensus.cc:740] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.407011 9978 consensus_queue.cc:260] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.419536 9978 ts_tablet_manager.cc:1434] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.023s user 0.005s sys 0.003s
I20260317 21:20:06.420773 9978 tablet_bootstrap.cc:492] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:06.433383 9278 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 08149c2a49274388be9400bdc390fab1
I20260317 21:20:06.435107 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 08149c2a49274388be9400bdc390fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41b86d09c802454dbbbb7fed1a3fc4bf) from {username='slave'} at 127.0.0.1:34012
W20260317 21:20:06.436614 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 08149c2a49274388be9400bdc390fab1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08149c2a49274388be9400bdc390fab1 already in progress: creating tablet
I20260317 21:20:06.437328 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 08149c2a49274388be9400bdc390fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41b86d09c802454dbbbb7fed1a3fc4bf) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.437661 9278 catalog_manager.cc:6741] ReplaceTablet: tablet 08149c2a49274388be9400bdc390fab1 deleted and replaced by tablet 41b86d09c802454dbbbb7fed1a3fc4bf
I20260317 21:20:06.438159 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet 08149c2a49274388be9400bdc390fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41b86d09c802454dbbbb7fed1a3fc4bf) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:06.439574 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet d1518f4f44504899a80615d9694c7e21 from {username='slave'} at 127.0.0.1:35054
W20260317 21:20:06.440717 9268 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 08149c2a49274388be9400bdc390fab1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 08149c2a49274388be9400bdc390fab1 already in progress: creating tablet
@ 0x7fd4ee3506bf google::LogMessage::~LogMessage() at ??:0
I20260317 21:20:06.445443 9976 tablet_bootstrap.cc:654] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.446866 9978 tablet_bootstrap.cc:654] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.447918 9973 tablet_bootstrap.cc:492] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:06.454219 9973 tablet_bootstrap.cc:654] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.459657 9979 ts_tablet_manager.cc:1929] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.473178 9979 log.cc:1199] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/d7f2f33fb3744ae0b8db16d529a9def2
I20260317 21:20:06.474486 9978 tablet_bootstrap.cc:492] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.474886 9978 ts_tablet_manager.cc:1403] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.005s
I20260317 21:20:06.477962 9978 raft_consensus.cc:359] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.478989 9978 raft_consensus.cc:385] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.479456 9978 raft_consensus.cc:740] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.480547 9976 tablet_bootstrap.cc:492] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:06.480248 9978 consensus_queue.cc:260] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.481031 9976 ts_tablet_manager.cc:1403] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.006s
I20260317 21:20:06.482806 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet d1518f4f44504899a80615d9694c7e21
I20260317 21:20:06.483471 9976 raft_consensus.cc:359] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.484166 9976 raft_consensus.cc:385] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.484786 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet d1518f4f44504899a80615d9694c7e21 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e8dac62e08450da8b7ad233ebfcdb4) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.485566 9982 tablet_replica.cc:333] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.486235 9982 raft_consensus.cc:2243] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.486732 9982 raft_consensus.cc:2272] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.489198 9982 ts_tablet_manager.cc:1916] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.494777 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet d1518f4f44504899a80615d9694c7e21 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e8dac62e08450da8b7ad233ebfcdb4) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.495558 9979 ts_tablet_manager.cc:1950] T d7f2f33fb3744ae0b8db16d529a9def2 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:06.495705 9981 tablet_replica.cc:333] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:06.496987 9981 raft_consensus.cc:2243] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.498404 9981 raft_consensus.cc:2272] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.501511 9981 ts_tablet_manager.cc:1916] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.502029 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet d7f2f33fb3744ae0b8db16d529a9def2 (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:06.498469 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet d1518f4f44504899a80615d9694c7e21 with delete_type TABLET_DATA_DELETED (replaced by tablet 70e8dac62e08450da8b7ad233ebfcdb4) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:06.498785 9973 tablet_bootstrap.cc:492] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:06.503067 9973 ts_tablet_manager.cc:1403] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.055s user 0.009s sys 0.004s
I20260317 21:20:06.503906 9277 catalog_manager.cc:6741] ReplaceTablet: tablet d1518f4f44504899a80615d9694c7e21 deleted and replaced by tablet 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:06.505615 9973 raft_consensus.cc:359] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.509943 9978 ts_tablet_manager.cc:1434] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.035s user 0.007s sys 0.000s
I20260317 21:20:06.511611 9844 tablet_service.cc:1558] Processing DeleteTablet for tablet 08149c2a49274388be9400bdc390fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41b86d09c802454dbbbb7fed1a3fc4bf) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.513370 9977 raft_consensus.cc:493] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:06.515787 9977 raft_consensus.cc:515] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.518095 9977 leader_election.cc:290] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595), 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:20:06.519263 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08149c2a49274388be9400bdc390fab1" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:06.520026 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08149c2a49274388be9400bdc390fab1" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
I20260317 21:20:06.520895 9865 raft_consensus.cc:2468] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 0.
I20260317 21:20:06.528782 9652 leader_election.cc:304] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:06.529703 9977 raft_consensus.cc:2804] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:06.530048 9977 raft_consensus.cc:493] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:06.530246 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.530364 9977 raft_consensus.cc:3060] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:06.531399 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b86d09c802454dbbbb7fed1a3fc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.531440 9978 tablet_bootstrap.cc:492] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:06.534225 9844 tablet_service.cc:1511] Processing CreateTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.535322 9844 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b86d09c802454dbbbb7fed1a3fc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.511313 9979 tablet_replica.cc:333] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:06.538683 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.540045 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 41b86d09c802454dbbbb7fed1a3fc4bf. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.507113 9973 raft_consensus.cc:385] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.541461 9973 raft_consensus.cc:740] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.484438 9976 raft_consensus.cc:740] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.550056 9976 consensus_queue.cc:260] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.543865 9973 consensus_queue.cc:260] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.555522 9977 raft_consensus.cc:515] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.556818 9979 raft_consensus.cc:2243] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.557294 9569 raft_consensus.cc:2468] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 0.
I20260317 21:20:06.559520 9979 raft_consensus.cc:2272] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.561616 9977 leader_election.cc:290] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Requested vote from peers 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595), 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:20:06.561918 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08149c2a49274388be9400bdc390fab1" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218"
I20260317 21:20:06.562518 9865 raft_consensus.cc:3060] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:06.563007 9568 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "08149c2a49274388be9400bdc390fab1" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4"
I20260317 21:20:06.567714 9978 tablet_bootstrap.cc:654] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
@ 0x7fd4f93fd06c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260317 21:20:06.571560 9865 raft_consensus.cc:2468] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 1.
I20260317 21:20:06.573038 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 08149c2a49274388be9400bdc390fab1 with delete_type TABLET_DATA_DELETED (replaced by tablet 41b86d09c802454dbbbb7fed1a3fc4bf) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.573449 9652 leader_election.cc:304] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters: 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:06.574465 9981 ts_tablet_manager.cc:1929] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.574958 9981 log.cc:1199] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/d1518f4f44504899a80615d9694c7e21
I20260317 21:20:06.575731 9981 ts_tablet_manager.cc:1950] T d1518f4f44504899a80615d9694c7e21 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:06.576876 9976 ts_tablet_manager.cc:1434] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.096s user 0.006s sys 0.002s
I20260317 21:20:06.577931 9976 tablet_bootstrap.cc:492] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.578900 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet d1518f4f44504899a80615d9694c7e21 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:06.581314 9981 tablet_replica.cc:333] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:06.582172 9981 raft_consensus.cc:2243] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.582665 9981 raft_consensus.cc:2272] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.585119 9973 ts_tablet_manager.cc:1434] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.082s user 0.006s sys 0.001s
W20260317 21:20:06.591862 9652 raft_consensus.cc:2686] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:20:06.597515 9976 tablet_bootstrap.cc:654] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.598482 9981 ts_tablet_manager.cc:1916] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.602667 9979 ts_tablet_manager.cc:1916] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.605639 9978 tablet_bootstrap.cc:492] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.606096 9978 ts_tablet_manager.cc:1403] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.075s user 0.010s sys 0.000s
I20260317 21:20:06.611681 9978 raft_consensus.cc:359] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.612345 9978 raft_consensus.cc:385] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.612608 9978 raft_consensus.cc:740] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.615129 9982 ts_tablet_manager.cc:1929] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.615545 9982 log.cc:1199] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/d1518f4f44504899a80615d9694c7e21
I20260317 21:20:06.616272 9982 ts_tablet_manager.cc:1950] T d1518f4f44504899a80615d9694c7e21 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.620544 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet d1518f4f44504899a80615d9694c7e21 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:06.623294 9982 tablet_replica.cc:333] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.624013 9982 raft_consensus.cc:2243] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.624559 9982 raft_consensus.cc:2272] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.627250 9547 tablet_service.cc:1511] Processing CreateTablet for tablet 70e8dac62e08450da8b7ad233ebfcdb4 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.628453 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e8dac62e08450da8b7ad233ebfcdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.629086 9973 tablet_bootstrap.cc:492] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:06.636236 9841 tablet_service.cc:1511] Processing CreateTablet for tablet 70e8dac62e08450da8b7ad233ebfcdb4 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.637070 9978 consensus_queue.cc:260] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.637599 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e8dac62e08450da8b7ad233ebfcdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.644977 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet b8519cc49ca34a5fa0941386938ea7fc from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:06.652053 9978 ts_tablet_manager.cc:1434] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.045s user 0.005s sys 0.000s
I20260317 21:20:06.660866 9978 tablet_bootstrap.cc:492] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:06.663345 9976 tablet_bootstrap.cc:492] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:06.663864 9976 ts_tablet_manager.cc:1403] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.086s user 0.012s sys 0.002s
I20260317 21:20:06.665719 9973 tablet_bootstrap.cc:654] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.670964 9976 raft_consensus.cc:359] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.682817 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet b8519cc49ca34a5fa0941386938ea7fc
I20260317 21:20:06.683943 9979 ts_tablet_manager.cc:1929] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.684304 9277 catalog_manager.cc:6741] ReplaceTablet: tablet b8519cc49ca34a5fa0941386938ea7fc deleted and replaced by tablet 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:06.684481 9976 raft_consensus.cc:385] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.684935 9976 raft_consensus.cc:740] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.685384 9979 log.cc:1199] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/08149c2a49274388be9400bdc390fab1
I20260317 21:20:06.686571 9979 ts_tablet_manager.cc:1950] T 08149c2a49274388be9400bdc390fab1 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:06.687043 9976 consensus_queue.cc:260] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.687775 9845 tablet_service.cc:1558] Processing DeleteTablet for tablet b8519cc49ca34a5fa0941386938ea7fc with delete_type TABLET_DATA_DELETED (replaced by tablet 889aad7114b04c30b394b3c9936b2abc) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.694476 9982 ts_tablet_manager.cc:1916] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.695168 9976 ts_tablet_manager.cc:1434] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.031s user 0.006s sys 0.004s
I20260317 21:20:06.701635 9979 tablet_replica.cc:333] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:06.702441 9979 raft_consensus.cc:2243] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.701169 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet b8519cc49ca34a5fa0941386938ea7fc with delete_type TABLET_DATA_DELETED (replaced by tablet 889aad7114b04c30b394b3c9936b2abc) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.707310 9981 ts_tablet_manager.cc:1929] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.707756 9981 log.cc:1199] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/08149c2a49274388be9400bdc390fab1
I20260317 21:20:06.708570 9981 ts_tablet_manager.cc:1950] T 08149c2a49274388be9400bdc390fab1 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:06.716475 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 41b86d09c802454dbbbb7fed1a3fc4bf from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:06.700435 9976 tablet_bootstrap.cc:492] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.702956 9979 raft_consensus.cc:2272] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.716533 9973 tablet_bootstrap.cc:492] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:06.723450 9973 ts_tablet_manager.cc:1403] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.095s user 0.011s sys 0.001s
I20260317 21:20:06.725044 9979 ts_tablet_manager.cc:1916] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.727144 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 08149c2a49274388be9400bdc390fab1 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.729604 9981 tablet_replica.cc:333] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:06.730129 9976 tablet_bootstrap.cc:654] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.735165 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 08149c2a49274388be9400bdc390fab1 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.737464 9973 raft_consensus.cc:359] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.737804 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet b8519cc49ca34a5fa0941386938ea7fc with delete_type TABLET_DATA_DELETED (replaced by tablet 889aad7114b04c30b394b3c9936b2abc) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:06.740262 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 889aad7114b04c30b394b3c9936b2abc (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.741595 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889aad7114b04c30b394b3c9936b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.742564 9697 tablet_service.cc:1511] Processing CreateTablet for tablet 889aad7114b04c30b394b3c9936b2abc (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.738157 9973 raft_consensus.cc:385] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.743355 9973 raft_consensus.cc:740] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.743803 9697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889aad7114b04c30b394b3c9936b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.744340 9973 consensus_queue.cc:260] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.742321 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 889aad7114b04c30b394b3c9936b2abc (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.763623 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 889aad7114b04c30b394b3c9936b2abc. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd4f93fc82c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260317 21:20:06.752609 9973 ts_tablet_manager.cc:1434] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.029s user 0.005s sys 0.000s
I20260317 21:20:06.781926 9981 raft_consensus.cc:2243] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.758710 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 70e8dac62e08450da8b7ad233ebfcdb4 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.789078 9976 tablet_bootstrap.cc:492] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:06.789628 9976 ts_tablet_manager.cc:1403] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.089s user 0.017s sys 0.000s
I20260317 21:20:06.782497 9981 raft_consensus.cc:2272] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.792120 9976 raft_consensus.cc:359] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.792886 9976 raft_consensus.cc:385] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.793155 9976 raft_consensus.cc:740] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.793895 9976 consensus_queue.cc:260] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.787731 9979 ts_tablet_manager.cc:1929] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.795243 9979 log.cc:1199] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/d1518f4f44504899a80615d9694c7e21
I20260317 21:20:06.795397 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 41b86d09c802454dbbbb7fed1a3fc4bf
I20260317 21:20:06.795955 9979 ts_tablet_manager.cc:1950] T d1518f4f44504899a80615d9694c7e21 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:06.792691 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 70e8dac62e08450da8b7ad233ebfcdb4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.797408 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf with delete_type TABLET_DATA_DELETED (replaced by tablet 37c9ed3e0fed4466bb60613cdc250c9e) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.797608 9277 catalog_manager.cc:6741] ReplaceTablet: tablet 41b86d09c802454dbbbb7fed1a3fc4bf deleted and replaced by tablet 37c9ed3e0fed4466bb60613cdc250c9e
I20260317 21:20:06.798831 9976 ts_tablet_manager.cc:1434] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.009s user 0.005s sys 0.000s
I20260317 21:20:06.800982 9979 tablet_replica.cc:333] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:06.801131 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf with delete_type TABLET_DATA_DELETED (replaced by tablet 37c9ed3e0fed4466bb60613cdc250c9e) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:06.801599 9978 tablet_bootstrap.cc:654] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.800714 9982 ts_tablet_manager.cc:1929] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.801673 9979 raft_consensus.cc:2243] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.801976 9982 log.cc:1199] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/08149c2a49274388be9400bdc390fab1
I20260317 21:20:06.802112 9979 raft_consensus.cc:2272] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.802642 9982 ts_tablet_manager.cc:1950] T 08149c2a49274388be9400bdc390fab1 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.803375 9976 tablet_bootstrap.cc:492] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.807751 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 08149c2a49274388be9400bdc390fab1 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.808779 9982 tablet_replica.cc:333] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.809425 9982 raft_consensus.cc:2243] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.809890 9982 raft_consensus.cc:2272] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.822984 9979 ts_tablet_manager.cc:1916] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.823957 9982 ts_tablet_manager.cc:1916] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.826192 9976 tablet_bootstrap.cc:654] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.829226 9696 tablet_service.cc:1558] Processing DeleteTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf with delete_type TABLET_DATA_DELETED (replaced by tablet 37c9ed3e0fed4466bb60613cdc250c9e) from {username='slave'} at 127.0.0.1:47636
@ 0x7fd4f94008a2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260317 21:20:06.839004 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet d1518f4f44504899a80615d9694c7e21 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:06.840404 9995 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:06.840960 9995 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:06.849661 9976 tablet_bootstrap.cc:492] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:06.850090 9976 ts_tablet_manager.cc:1403] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.047s user 0.013s sys 0.000s
I20260317 21:20:06.852288 9982 ts_tablet_manager.cc:1929] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.853067 9982 log.cc:1199] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/b8519cc49ca34a5fa0941386938ea7fc
I20260317 21:20:06.853767 9982 ts_tablet_manager.cc:1950] T b8519cc49ca34a5fa0941386938ea7fc P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.853497 9976 raft_consensus.cc:359] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.854153 9976 raft_consensus.cc:385] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.854424 9976 raft_consensus.cc:740] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.854658 9995 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:06.855141 9976 consensus_queue.cc:260] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.856573 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet b8519cc49ca34a5fa0941386938ea7fc (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:06.858603 9979 ts_tablet_manager.cc:1929] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.859074 9979 log.cc:1199] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/b8519cc49ca34a5fa0941386938ea7fc
I20260317 21:20:06.859930 9979 ts_tablet_manager.cc:1950] T b8519cc49ca34a5fa0941386938ea7fc P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:06.862375 9979 tablet_replica.cc:333] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:06.863005 9979 raft_consensus.cc:2243] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.863449 9979 raft_consensus.cc:2272] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.863682 9982 tablet_replica.cc:333] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:06.864353 9982 raft_consensus.cc:2243] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:06.872556 9976 ts_tablet_manager.cc:1434] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260317 21:20:06.872902 9982 raft_consensus.cc:2272] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:06.875180 9982 ts_tablet_manager.cc:1916] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.877655 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet b8519cc49ca34a5fa0941386938ea7fc (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:06.878443 9976 tablet_bootstrap.cc:492] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:06.884357 9976 tablet_bootstrap.cc:654] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.885787 9973 tablet_bootstrap.cc:492] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:06.888440 9979 ts_tablet_manager.cc:1916] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.891156 9981 ts_tablet_manager.cc:1916] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:06.891793 9973 tablet_bootstrap.cc:654] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.895922 9978 tablet_bootstrap.cc:492] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:06.896387 9978 ts_tablet_manager.cc:1403] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.236s user 0.009s sys 0.000s
@ 0x7fd4f940085a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260317 21:20:06.899132 9976 tablet_bootstrap.cc:492] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:06.899708 9976 ts_tablet_manager.cc:1403] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.003s
I20260317 21:20:06.902045 9976 raft_consensus.cc:359] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.902676 9976 raft_consensus.cc:385] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.902908 9976 raft_consensus.cc:740] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.903774 9976 consensus_queue.cc:260] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.910567 9982 ts_tablet_manager.cc:1929] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.910972 9982 log.cc:1199] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/41b86d09c802454dbbbb7fed1a3fc4bf
I20260317 21:20:06.911690 9982 ts_tablet_manager.cc:1950] T 41b86d09c802454dbbbb7fed1a3fc4bf P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:06.915910 9976 ts_tablet_manager.cc:1434] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.016s user 0.007s sys 0.000s
I20260317 21:20:06.916066 9973 tablet_bootstrap.cc:492] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:06.916633 9973 ts_tablet_manager.cc:1403] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.010s
I20260317 21:20:06.918874 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 70e8dac62e08450da8b7ad233ebfcdb4 from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:06.919034 9973 raft_consensus.cc:359] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.919713 9973 raft_consensus.cc:385] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.919970 9973 raft_consensus.cc:740] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.923854 9973 consensus_queue.cc:260] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.930090 9979 ts_tablet_manager.cc:1929] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:06.930943 9979 log.cc:1199] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/41b86d09c802454dbbbb7fed1a3fc4bf
I20260317 21:20:06.931707 9979 ts_tablet_manager.cc:1950] T 41b86d09c802454dbbbb7fed1a3fc4bf P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:06.938911 9973 ts_tablet_manager.cc:1434] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.022s user 0.005s sys 0.000s
I20260317 21:20:06.943846 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 41b86d09c802454dbbbb7fed1a3fc4bf (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.944888 9973 tablet_bootstrap.cc:492] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:06.945075 9270 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 41b86d09c802454dbbbb7fed1a3fc4bf (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:06.951148 9973 tablet_bootstrap.cc:654] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:06.900031 9978 raft_consensus.cc:359] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.956861 9978 raft_consensus.cc:385] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.957159 9978 raft_consensus.cc:740] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.960067 9978 consensus_queue.cc:260] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.967389 9547 tablet_service.cc:1511] Processing CreateTablet for tablet 37c9ed3e0fed4466bb60613cdc250c9e (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:20:06.967886 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:06.968667 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c9ed3e0fed4466bb60613cdc250c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.969261 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 37c9ed3e0fed4466bb60613cdc250c9e (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:06.970583 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c9ed3e0fed4466bb60613cdc250c9e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:06.973976 9845 tablet_service.cc:1511] Processing CreateTablet for tablet 37c9ed3e0fed4466bb60613cdc250c9e (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7fd4f94007ea std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260317 21:20:06.982762 9978 ts_tablet_manager.cc:1434] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.086s user 0.005s sys 0.000s
I20260317 21:20:06.983706 9978 tablet_bootstrap.cc:492] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:06.985989 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 41b86d09c802454dbbbb7fed1a3fc4bf with delete_type TABLET_DATA_DELETED (replaced by tablet 37c9ed3e0fed4466bb60613cdc250c9e) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:06.986635 9973 tablet_bootstrap.cc:492] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:06.987066 9973 ts_tablet_manager.cc:1403] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.047s user 0.002s sys 0.008s
W20260317 21:20:06.989358 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:06.990026 9973 raft_consensus.cc:359] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.990693 9973 raft_consensus.cc:385] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:06.990972 9973 raft_consensus.cc:740] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:06.990542 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:06.991650 9973 consensus_queue.cc:260] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:06.993482 9995 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:20:06.996116 9268 catalog_manager.cc:4969] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 41b86d09c802454dbbbb7fed1a3fc4bf because the tablet was not found. No further retry: Not found: Tablet not found: 41b86d09c802454dbbbb7fed1a3fc4bf
I20260317 21:20:06.997581 9973 ts_tablet_manager.cc:1434] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20260317 21:20:07.001197 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37c9ed3e0fed4466bb60613cdc250c9e. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd4f94007b2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260317 21:20:07.030632 10017 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:07.031203 10017 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.036341 10017 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:07.036806 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e8dac62e08450da8b7ad233ebfcdb4" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:07.037479 9569 raft_consensus.cc:2468] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 0.
I20260317 21:20:07.037963 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e8dac62e08450da8b7ad233ebfcdb4" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
@ 0x7fd4f93ffaae std::__1::__function::__func<>::operator()() at ??:0
I20260317 21:20:07.038780 9651 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923; no voters:
W20260317 21:20:07.039768 9652 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Illegal state: must be running to vote when last-logged opid is not known
@ 0x5ab085 std::__1::__function::__value_func<>::operator()() at ??:?
I20260317 21:20:07.040077 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:07.041694 10017 raft_consensus.cc:2804] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:07.042150 9277 catalog_manager.cc:6741] ReplaceTablet: tablet 70e8dac62e08450da8b7ad233ebfcdb4 deleted and replaced by tablet ce91b45480ed48a78b76f6936f7a6806
I20260317 21:20:07.042132 10017 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:07.042500 10017 raft_consensus.cc:3060] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
@ 0x5aaeba std::__1::function<>::operator()() at ??:?
I20260317 21:20:07.043314 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 70e8dac62e08450da8b7ad233ebfcdb4 with delete_type TABLET_DATA_DELETED (replaced by tablet ce91b45480ed48a78b76f6936f7a6806) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:07.044878 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet 70e8dac62e08450da8b7ad233ebfcdb4 with delete_type TABLET_DATA_DELETED (replaced by tablet ce91b45480ed48a78b76f6936f7a6806) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:07.045799 9979 tablet_replica.cc:333] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:07.047529 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 70e8dac62e08450da8b7ad233ebfcdb4 with delete_type TABLET_DATA_DELETED (replaced by tablet ce91b45480ed48a78b76f6936f7a6806) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:07.048470 9982 tablet_replica.cc:333] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:07.051270 9982 raft_consensus.cc:2243] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:07.051826 9982 raft_consensus.cc:2272] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:07.053740 9978 tablet_bootstrap.cc:654] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.055773 9982 ts_tablet_manager.cc:1916] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:07.060963 9973 tablet_bootstrap.cc:492] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:07.063923 9981 ts_tablet_manager.cc:1929] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:07.064308 9981 log.cc:1199] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/b8519cc49ca34a5fa0941386938ea7fc
I20260317 21:20:07.064985 9981 ts_tablet_manager.cc:1950] T b8519cc49ca34a5fa0941386938ea7fc P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:07.067556 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet b8519cc49ca34a5fa0941386938ea7fc (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]) successfully deleted
I20260317 21:20:07.069382 9973 tablet_bootstrap.cc:654] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.075945 9277 master_service.cc:954] ReplaceTablet: received request to replace tablet 889aad7114b04c30b394b3c9936b2abc from {username='slave'} at 127.0.0.1:35054
I20260317 21:20:07.087138 9981 tablet_replica.cc:333] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:07.087911 9981 raft_consensus.cc:2243] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:07.089007 9981 raft_consensus.cc:2272] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:07.098049 9973 tablet_bootstrap.cc:492] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:07.099256 9973 ts_tablet_manager.cc:1403] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.038s user 0.011s sys 0.002s
I20260317 21:20:07.101956 9973 raft_consensus.cc:359] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.102639 9973 raft_consensus.cc:385] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.102895 9973 raft_consensus.cc:740] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.106962 9973 consensus_queue.cc:260] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.107551 9277 catalog_manager.cc:5958] T 00000000000000000000000000000000 P a4719f316c614a9cadaf5edab834804d: Sending DeleteTablet for 3 replicas of tablet 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:07.109930 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 889aad7114b04c30b394b3c9936b2abc with delete_type TABLET_DATA_DELETED (replaced by tablet 841735a56c4c4df7a58f93ddbaf67a09) from {username='slave'} at 127.0.0.1:47636
I20260317 21:20:07.110562 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 889aad7114b04c30b394b3c9936b2abc with delete_type TABLET_DATA_DELETED (replaced by tablet 841735a56c4c4df7a58f93ddbaf67a09) from {username='slave'} at 127.0.0.1:34012
I20260317 21:20:07.111382 9277 catalog_manager.cc:6741] ReplaceTablet: tablet 889aad7114b04c30b394b3c9936b2abc deleted and replaced by tablet 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:07.113425 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 889aad7114b04c30b394b3c9936b2abc with delete_type TABLET_DATA_DELETED (replaced by tablet 841735a56c4c4df7a58f93ddbaf67a09) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:07.114029 9982 ts_tablet_manager.cc:1929] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:07.114391 9982 log.cc:1199] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:07.114918 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 889aad7114b04c30b394b3c9936b2abc with delete_type TABLET_DATA_DELETED (replaced by tablet 841735a56c4c4df7a58f93ddbaf67a09) from {username='slave'} at 127.0.0.1:33304
I20260317 21:20:07.115058 9982 ts_tablet_manager.cc:1950] T 70e8dac62e08450da8b7ad233ebfcdb4 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
@ 0x7fd4f057274b kudu::Thread::SuperviseThread() at ??:0
@ 0x4749c0 __tsan_thread_start_func at ??:?
I20260317 21:20:07.124936 9973 ts_tablet_manager.cc:1434] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.025s user 0.008s sys 0.001s
I20260317 21:20:07.127291 9978 tablet_bootstrap.cc:492] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:07.127779 9978 ts_tablet_manager.cc:1403] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.144s user 0.005s sys 0.008s
I20260317 21:20:07.133991 9268 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 70e8dac62e08450da8b7ad233ebfcdb4 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:07.136783 9982 tablet_replica.cc:333] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:07.137400 9982 raft_consensus.cc:2243] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:07.137871 9982 raft_consensus.cc:2272] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:07.142334 9981 ts_tablet_manager.cc:1916] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:07.150471 9982 ts_tablet_manager.cc:1916] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:07.153225 9978 raft_consensus.cc:359] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.153888 9978 raft_consensus.cc:385] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.154127 9978 raft_consensus.cc:740] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.157126 9976 tablet_bootstrap.cc:492] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:07.113073 10017 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.158540 9979 raft_consensus.cc:2243] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:07.159518 9979 raft_consensus.cc:2272] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:07.160174 9978 consensus_queue.cc:260] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
W20260317 21:20:07.161692 9972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260317 21:20:07.162631 9976 tablet_bootstrap.cc:654] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.171106 9698 tablet_service.cc:1511] Processing CreateTablet for tablet ce91b45480ed48a78b76f6936f7a6806 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:07.172513 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce91b45480ed48a78b76f6936f7a6806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:07.172941 9845 tablet_service.cc:1511] Processing CreateTablet for tablet ce91b45480ed48a78b76f6936f7a6806 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:07.174198 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce91b45480ed48a78b76f6936f7a6806. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd4f1c7d6db start_thread at ??:0
W20260317 21:20:07.177348 9651 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:07.180744 9278 catalog_manager.cc:5504] Ignoring report from unknown tablet 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:07.185410 9972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:20:07.186197 9972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260317 21:20:07.193071 9981 ts_tablet_manager.cc:1929] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:07.193678 9981 log.cc:1199] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/41b86d09c802454dbbbb7fed1a3fc4bf
I20260317 21:20:07.194336 9981 ts_tablet_manager.cc:1950] T 41b86d09c802454dbbbb7fed1a3fc4bf P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:07.195065 9982 ts_tablet_manager.cc:1929] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:07.195415 9982 log.cc:1199] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:07.196082 9982 ts_tablet_manager.cc:1950] T 889aad7114b04c30b394b3c9936b2abc P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:07.196569 9269 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 41b86d09c802454dbbbb7fed1a3fc4bf (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:07.208982 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "70e8dac62e08450da8b7ad233ebfcdb4" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218"
I20260317 21:20:07.209821 9865 raft_consensus.cc:3060] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.210012 9697 tablet_service.cc:1511] Processing CreateTablet for tablet 841735a56c4c4df7a58f93ddbaf67a09 (DEFAULT_TABLE table=default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]), partition=RANGE (key) PARTITION UNBOUNDED
W20260317 21:20:07.212981 9269 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 70e8dac62e08450da8b7ad233ebfcdb4 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 70e8dac62e08450da8b7ad233ebfcdb4 already in progress: creating tablet
I20260317 21:20:07.212634 9697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 841735a56c4c4df7a58f93ddbaf67a09. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7fd4ecff561f clone at ??:0
I20260317 21:20:07.218727 9865 raft_consensus.cc:2468] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 1.
I20260317 21:20:07.220362 9652 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters: 9b9886349405418f80c5511b013ffbd4
W20260317 21:20:07.221338 9652 raft_consensus.cc:2686] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: unable to run election callback: Service unavailable: Thread pool token was shut down
I20260317 21:20:07.224335 9978 ts_tablet_manager.cc:1434] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.096s user 0.007s sys 0.000s
I20260317 21:20:07.225132 9973 tablet_bootstrap.cc:492] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:07.225076 10017 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:07.230589 9973 tablet_bootstrap.cc:654] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.231772 9979 ts_tablet_manager.cc:1916] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:07.235076 9978 tablet_bootstrap.cc:492] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:07.241565 9547 tablet_service.cc:1511] Processing CreateTablet for tablet ce91b45480ed48a78b76f6936f7a6806 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:07.242743 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce91b45480ed48a78b76f6936f7a6806. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:07.250545 9976 tablet_bootstrap.cc:492] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:07.250949 9976 ts_tablet_manager.cc:1403] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.094s user 0.009s sys 0.022s
I20260317 21:20:07.251894 9978 tablet_bootstrap.cc:654] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.253799 9976 raft_consensus.cc:359] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.254443 9976 raft_consensus.cc:385] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.254716 9976 raft_consensus.cc:740] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.256322 10017 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:07.256807 10017 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.258953 10017 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:07.262341 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "889aad7114b04c30b394b3c9936b2abc" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
W20260317 21:20:07.263877 9652 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Illegal state: must be running to vote when last-logged opid is not known
W20260317 21:20:07.265225 9651 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:07.265074 9976 consensus_queue.cc:260] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.265638 9651 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:07.267614 10017 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:07.271497 9973 tablet_bootstrap.cc:492] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:07.272029 9973 ts_tablet_manager.cc:1403] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.047s user 0.010s sys 0.004s
I20260317 21:20:07.275098 9979 ts_tablet_manager.cc:1929] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:07.275643 9979 log.cc:1199] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:07.276598 9979 ts_tablet_manager.cc:1950] T 70e8dac62e08450da8b7ad233ebfcdb4 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:07.287539 9979 tablet_replica.cc:333] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:07.290120 9973 raft_consensus.cc:359] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.290912 9973 raft_consensus.cc:385] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.291167 9973 raft_consensus.cc:740] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.288261 9979 raft_consensus.cc:2243] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:07.292800 9978 tablet_bootstrap.cc:492] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:07.292477 9973 consensus_queue.cc:260] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.293025 9979 raft_consensus.cc:2272] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:07.293277 9978 ts_tablet_manager.cc:1403] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.058s user 0.009s sys 0.011s
I20260317 21:20:07.295815 9973 ts_tablet_manager.cc:1434] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.024s user 0.011s sys 0.000s
I20260317 21:20:07.298082 9978 raft_consensus.cc:359] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.298799 9978 raft_consensus.cc:385] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.299045 9978 raft_consensus.cc:740] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.299728 9978 consensus_queue.cc:260] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.302028 9978 ts_tablet_manager.cc:1434] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.008s user 0.006s sys 0.000s
I20260317 21:20:07.308686 9979 ts_tablet_manager.cc:1916] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:07.313462 9976 ts_tablet_manager.cc:1434] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.062s user 0.007s sys 0.001s
I20260317 21:20:07.314839 9976 tablet_bootstrap.cc:492] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:07.321187 9978 tablet_bootstrap.cc:492] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:07.321496 9976 tablet_bootstrap.cc:654] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.324615 9973 tablet_bootstrap.cc:492] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:07.332106 9978 tablet_bootstrap.cc:654] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.340340 9973 tablet_bootstrap.cc:654] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:07.341524 9976 tablet_bootstrap.cc:492] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:07.341925 9976 ts_tablet_manager.cc:1403] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.027s user 0.013s sys 0.000s
I20260317 21:20:07.344451 9976 raft_consensus.cc:359] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.345165 9976 raft_consensus.cc:385] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.345444 9976 raft_consensus.cc:740] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.346061 9976 consensus_queue.cc:260] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.352268 9976 ts_tablet_manager.cc:1434] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.010s user 0.007s sys 0.000s
I20260317 21:20:07.366966 9973 tablet_bootstrap.cc:492] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:07.367357 9978 tablet_bootstrap.cc:492] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:07.367455 9973 ts_tablet_manager.cc:1403] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.043s user 0.010s sys 0.000s
I20260317 21:20:07.367769 9978 ts_tablet_manager.cc:1403] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.047s user 0.008s sys 0.002s
I20260317 21:20:07.368031 9979 ts_tablet_manager.cc:1929] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:07.368492 9979 log.cc:1199] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:07.370184 9973 raft_consensus.cc:359] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.370472 9978 raft_consensus.cc:359] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.370761 9979 ts_tablet_manager.cc:1950] T 889aad7114b04c30b394b3c9936b2abc P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:07.371068 9978 raft_consensus.cc:385] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.370985 9973 raft_consensus.cc:385] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.371313 9973 raft_consensus.cc:740] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.371295 9978 raft_consensus.cc:740] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.371963 9978 consensus_queue.cc:260] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.372910 9973 consensus_queue.cc:260] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.385334 9978 ts_tablet_manager.cc:1434] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.017s user 0.007s sys 0.000s
I20260317 21:20:07.393198 9973 ts_tablet_manager.cc:1434] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.025s user 0.008s sys 0.000s
I20260317 21:20:07.398852 9978 tablet_bootstrap.cc:492] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:07.404428 9978 tablet_bootstrap.cc:654] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
W20260317 21:20:07.435003 9972 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260317 21:20:07.435354 9972 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260317 21:20:07.435669 9972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:20:07.435977 9972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:20:07.436328 9972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:20:07.436721 9972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:20:07.442858 9978 tablet_bootstrap.cc:492] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:07.443300 9978 ts_tablet_manager.cc:1403] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.000s
I20260317 21:20:07.445709 9978 raft_consensus.cc:359] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.446362 9978 raft_consensus.cc:385] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:07.446638 9978 raft_consensus.cc:740] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:07.447235 9978 consensus_queue.cc:260] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.449388 9978 ts_tablet_manager.cc:1434] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.006s user 0.005s sys 0.000s
I20260317 21:20:07.508131 10017 raft_consensus.cc:493] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:07.508673 10017 raft_consensus.cc:515] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.510538 10017 leader_election.cc:290] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:07.512115 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce91b45480ed48a78b76f6936f7a6806" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
I20260317 21:20:07.512758 9865 raft_consensus.cc:2468] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 0.
I20260317 21:20:07.513760 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce91b45480ed48a78b76f6936f7a6806" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:07.514587 9569 raft_consensus.cc:2468] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 0.
I20260317 21:20:07.515846 9652 leader_election.cc:304] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:07.516613 10017 raft_consensus.cc:2804] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:07.516966 10017 raft_consensus.cc:493] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:07.517272 10017 raft_consensus.cc:3060] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.519109 9972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
--hive_metastore_uris=thrift://127.0.0.1:42187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.124:41015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.6.122.124
--webserver_port=33267
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:20:07.520885 9972 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:20:07.523581 9972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20260317 21:20:07.524626 10017 raft_consensus.cc:515] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.526686 10017 leader_election.cc:290] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:07.528513 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce91b45480ed48a78b76f6936f7a6806" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218"
I20260317 21:20:07.529148 9865 raft_consensus.cc:3060] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.533757 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ce91b45480ed48a78b76f6936f7a6806" candidate_uuid: "f738662691924b8a8f9b4b7cf1804923" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4"
I20260317 21:20:07.534304 9569 raft_consensus.cc:3060] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.549994 9569 raft_consensus.cc:2468] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 1.
I20260317 21:20:07.554051 9865 raft_consensus.cc:2468] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f738662691924b8a8f9b4b7cf1804923 in term 1.
I20260317 21:20:07.555728 9652 leader_election.cc:304] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:07.558804 10017 raft_consensus.cc:2804] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:20:07.559314 10017 raft_consensus.cc:697] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 1 LEADER]: Becoming Leader. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Running, Role: LEADER
I20260317 21:20:07.560101 10017 consensus_queue.cc:237] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
W20260317 21:20:07.563261 10060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:07.591193 9995 raft_consensus.cc:493] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:07.591703 9995 raft_consensus.cc:515] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.593909 9995 leader_election.cc:290] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:07.596174 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c9ed3e0fed4466bb60613cdc250c9e" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:07.596843 9569 raft_consensus.cc:2468] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 0.
I20260317 21:20:07.596700 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c9ed3e0fed4466bb60613cdc250c9e" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923" is_pre_election: true
I20260317 21:20:07.597483 9719 raft_consensus.cc:2468] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 0.
I20260317 21:20:07.598285 9798 leader_election.cc:304] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4; no voters:
I20260317 21:20:07.599481 9995 raft_consensus.cc:2804] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:07.599903 9995 raft_consensus.cc:493] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:07.600310 9995 raft_consensus.cc:3060] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.610562 9995 raft_consensus.cc:515] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.614629 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c9ed3e0fed4466bb60613cdc250c9e" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4"
I20260317 21:20:07.615190 9569 raft_consensus.cc:3060] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.622457 9569 raft_consensus.cc:2468] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 1.
I20260317 21:20:07.624663 9798 leader_election.cc:304] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4; no voters:
I20260317 21:20:07.626335 10066 raft_consensus.cc:2804] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:20:07.628149 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37c9ed3e0fed4466bb60613cdc250c9e" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923"
I20260317 21:20:07.628969 9719 raft_consensus.cc:3060] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:07.629693 10066 raft_consensus.cc:697] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Running, Role: LEADER
I20260317 21:20:07.630853 10066 consensus_queue.cc:237] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:07.641709 9719 raft_consensus.cc:2468] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 1.
I20260317 21:20:07.642201 9995 leader_election.cc:290] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 election: Requested vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:07.718286 10069 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4719f316c614a9cadaf5edab834804d)
I20260317 21:20:07.718818 10069 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:07.732808 10069 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
W20260317 21:20:07.764472 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015): Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
W20260317 21:20:07.773145 9798 connection.cc:537] server connection from 127.0.0.1:34012 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.774062 9922 connection.cc:537] client connection to 127.6.122.125:38261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.775993 9922 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.125:38261)
W20260317 21:20:07.777630 9922 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.122.125:38261)
W20260317 21:20:07.778657 9922 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.122.125:38261)
W20260317 21:20:07.784302 9922 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.122.125:38261)
W20260317 21:20:07.787361 9800 connection.cc:537] client connection to 127.6.122.125:38261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.792299 9653 connection.cc:537] client connection to 127.6.122.125:38261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.793270 9653 connection.cc:537] server connection from 127.0.0.1:47636 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.794922 9503 connection.cc:537] client connection to 127.6.122.125:38261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.798820 9501 connection.cc:537] server connection from 127.0.0.1:33304 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.800033 9195 connection.cc:537] client connection to 127.6.122.125:38261 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:07.805179 9988 negotiation.cc:337] Failed RPC negotiation. Trace:
0317 21:20:07.789169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.122.125:38261 (local address 127.6.122.67:37487)
0317 21:20:07.790417 (+ 1248us) negotiation.cc:107] Waiting for socket to connect
0317 21:20:07.790494 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0317 21:20:07.790776 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0317 21:20:07.804040 (+ 13264us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":580}
W20260317 21:20:07.809826 9992 negotiation.cc:337] Failed RPC negotiation. Trace:
0317 21:20:07.794211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.122.125:38261 (local address 127.6.122.66:54981)
0317 21:20:07.794969 (+ 758us) negotiation.cc:107] Waiting for socket to connect
0317 21:20:07.795017 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0317 21:20:07.795278 (+ 261us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0317 21:20:07.808617 (+ 13339us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":88}
W20260317 21:20:07.813717 9970 negotiation.cc:337] Failed RPC negotiation. Trace:
0317 21:20:07.797016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.122.125:38261 (local address 127.6.122.65:42651)
0317 21:20:07.797918 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0317 21:20:07.797963 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0317 21:20:07.798308 (+ 345us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0317 21:20:07.811416 (+ 13108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":321}
W20260317 21:20:07.816845 10070 negotiation.cc:337] Failed RPC negotiation. Trace:
0317 21:20:07.803042 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.122.125:38261 (local address 127.0.0.1:34100)
0317 21:20:07.803853 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0317 21:20:07.803904 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0317 21:20:07.804354 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0317 21:20:07.815659 (+ 11305us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":102}
W20260317 21:20:07.817579 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260317 21:20:07.818007 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:07.820724 10069 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260317 21:20:07.844899 10072 negotiation.cc:337] Failed RPC negotiation. Trace:
0317 21:20:07.786624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.6.122.125:38261 (local address 127.0.0.1:34088)
0317 21:20:07.839604 (+ 52980us) negotiation.cc:107] Waiting for socket to connect
0317 21:20:07.839906 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":51438,"thread_start_us":50187,"threads_started":1}
I20260317 21:20:08.044030 10066 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:08.044502 10066 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:08.046171 10066 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:08.048228 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:08.048848 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:08.049388 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:08.050346 10066 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:08.194990 10074 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:08.196033 10074 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
W20260317 21:20:08.201766 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:08.203617 10074 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:08.205013 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:08.205829 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:08.208072 10074 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:08.427044 10074 consensus_queue.cc:1048] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:20:08.513974 10074 consensus_queue.cc:1048] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:20:08.555366 10084 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4719f316c614a9cadaf5edab834804d)
I20260317 21:20:08.555972 10084 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:08.558351 10084 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
W20260317 21:20:08.569914 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:08.570706 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015): Network error: Client connection negotiation failed: client connection to 127.6.122.124:41015: connect: Connection refused (error 111)
I20260317 21:20:08.571173 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d6b9687775954282a11f2cf829d24bac; no voters: a4719f316c614a9cadaf5edab834804d, ed19cc1cc9e74a0d9b8dc4a609f1bea9
I20260317 21:20:08.572415 10084 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260317 21:20:08.666586 10066 consensus_queue.cc:1048] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:20:08.724435 10089 consensus_queue.cc:1048] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260317 21:20:07.563443 10061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:20:08.956729 10059 debug-util.cc:398] Leaking SignalData structure 0x7b0800037b00 after lost signal to thread 9972
W20260317 21:20:09.195861 9972 thread.cc:641] OpenStack (cloud detector) Time spent creating pthread: real 1.633s user 0.516s sys 0.940s
W20260317 21:20:09.196327 9972 thread.cc:608] OpenStack (cloud detector) Time spent starting thread: real 1.633s user 0.516s sys 0.940s
W20260317 21:20:09.199386 10063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:09.205323 9972 server_base.cc:1060] running on GCE node
I20260317 21:20:09.206686 9972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:20:09.209699 9972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:20:09.211215 9972 hybrid_clock.cc:648] HybridClock initialized: now 1773782409211173 us; error 46 us; skew 500 ppm
I20260317 21:20:09.212152 9972 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:20:09.219352 9972 webserver.cc:492] Webserver started at http://127.6.122.124:33267/ using document root <none> and password file <none>
I20260317 21:20:09.221063 9972 fs_manager.cc:362] Metadata directory not provided
I20260317 21:20:09.221417 9972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:20:09.230556 9972 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.002s sys 0.002s
I20260317 21:20:09.236433 10102 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:09.237704 9972 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260317 21:20:09.238087 9972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9"
format_stamp: "Formatted at 2026-03-17 21:19:54 on dist-test-slave-979h"
I20260317 21:20:09.240157 9972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:20:09.343853 9972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:20:09.345665 9972 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:20:09.346184 9972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:20:09.443965 9972 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.124:41015
I20260317 21:20:09.444043 10159 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.124:41015 every 8 connection(s)
I20260317 21:20:09.447285 9972 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-2/data/info.pb
I20260317 21:20:09.448766 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 9972
I20260317 21:20:09.464695 10160 sys_catalog.cc:263] Verifying existing consensus state
I20260317 21:20:09.473965 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Bootstrap starting.
I20260317 21:20:09.497184 10160 log.cc:826] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Log is configured to *not* fsync() on all Append() calls
I20260317 21:20:09.515828 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 21:20:09.516716 10160 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Bootstrap complete.
I20260317 21:20:09.537652 10160 raft_consensus.cc:359] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:09.541316 10160 raft_consensus.cc:740] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ed19cc1cc9e74a0d9b8dc4a609f1bea9, State: Initialized, Role: FOLLOWER
I20260317 21:20:09.544011 10160 consensus_queue.cc:260] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:09.547448 10168 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:09.548200 10168 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:09.550087 10160 sys_catalog.cc:565] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:20:09.554687 9911 heartbeater.cc:344] Connected to a master server at 127.6.122.124:41015
I20260317 21:20:09.555222 9911 heartbeater.cc:499] Master 127.6.122.124:41015 was elected leader, sending a full tablet report...
I20260317 21:20:09.612015 10182 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:20:09.612335 10182 catalog_manager.cc:1562] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: loading cluster ID for follower catalog manager: success
I20260317 21:20:09.625423 10182 catalog_manager.cc:1584] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: acquiring CA information for follower catalog manager: success
I20260317 21:20:09.632483 10182 catalog_manager.cc:1612] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260317 21:20:09.638033 9972 master_runner.cc:186] Error getting master registration for 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111),
I20260317 21:20:09.666646 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:09.692418 10187 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting pre-election (detected failure of leader a4719f316c614a9cadaf5edab834804d)
I20260317 21:20:09.693143 10187 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:09.694938 10187 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
W20260317 21:20:09.713239 9195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111) [suppressed 4 similar messages]
I20260317 21:20:09.721184 10135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:09.722325 10135 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b9687775954282a11f2cf829d24bac in term 1.
W20260317 21:20:09.723567 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:09.724583 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d
I20260317 21:20:09.725454 10187 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260317 21:20:09.725811 10187 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Starting leader election (detected failure of leader a4719f316c614a9cadaf5edab834804d)
I20260317 21:20:09.726173 10187 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 2 FOLLOWER]: Advancing to term 3
I20260317 21:20:09.744755 10187 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:09.748752 9765 heartbeater.cc:344] Connected to a master server at 127.6.122.124:41015
I20260317 21:20:09.750502 10187 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 election: Requested vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:09.753490 10135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 3 candidate_status { last_received { term: 2 index: 34 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9"
I20260317 21:20:09.754060 10135 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 1 FOLLOWER]: Advancing to term 3
W20260317 21:20:09.758553 9195 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:09.767592 10135 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b9687775954282a11f2cf829d24bac in term 3.
I20260317 21:20:09.771238 9195 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d
I20260317 21:20:09.772966 10187 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Leader election won for term 3
I20260317 21:20:09.783504 10124 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" instance_seqno: 1773782401247982) as {username='slave'} at 127.6.122.66:40103; Asking this server to re-register.
I20260317 21:20:09.788400 9765 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:09.793044 10187 raft_consensus.cc:697] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 LEADER]: Becoming Leader. State: Replica: d6b9687775954282a11f2cf829d24bac, State: Running, Role: LEADER
I20260317 21:20:09.794740 10123 ts_manager.cc:194] Registered new tserver with Master: f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:09.794281 10187 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 33, Committed index: 33, Last appended: 2.34, Last appended by leader: 34, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:09.807708 10194 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b9687775954282a11f2cf829d24bac. Latest consensus state: current_term: 3 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:09.808682 10194 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: LEADER
I20260317 21:20:09.815562 9616 heartbeater.cc:344] Connected to a master server at 127.6.122.124:41015
I20260317 21:20:09.820289 10123 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9b9886349405418f80c5511b013ffbd4" instance_seqno: 1773782398616276) as {username='slave'} at 127.6.122.65:39667; Asking this server to re-register.
I20260317 21:20:09.822093 9616 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:09.825922 10123 ts_manager.cc:194] Registered new tserver with Master: 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:20:10.063987 9911 heartbeater.cc:499] Master 127.6.122.124:41015 was elected leader, sending a full tablet report...
I20260317 21:20:10.069205 10123 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6c2dd7c5a524427da691785d88218218" instance_seqno: 1773782403703098) as {username='slave'} at 127.6.122.67:45611; Asking this server to re-register.
I20260317 21:20:10.071035 9911 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:10.074723 10123 ts_manager.cc:194] Registered new tserver with Master: 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:10.350551 10135 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Refusing update from remote peer d6b9687775954282a11f2cf829d24bac: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 3 index: 35. (index mismatch)
I20260317 21:20:10.352061 10200 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 35, Last known committed idx: 4, Time since last communication: 0.000s
W20260317 21:20:10.366323 9195 consensus_peers.cc:597] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac -> Peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Couldn't send request to peer a4719f316c614a9cadaf5edab834804d. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:20:10.409746 10202 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b9687775954282a11f2cf829d24bac. Latest consensus state: current_term: 3 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:10.410502 10202 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:10.412420 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:10.413157 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:10.415314 10206 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:10.416013 10206 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:10.453001 10210 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:10.453977 10210 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: LEADER
I20260317 21:20:10.467763 10198 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:20:10.486627 10198 catalog_manager.cc:679] Loaded metadata for table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]
I20260317 21:20:10.490535 10198 catalog_manager.cc:679] Loaded metadata for table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]
I20260317 21:20:10.493840 10198 catalog_manager.cc:679] Loaded metadata for table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]
I20260317 21:20:10.509295 10198 tablet_loader.cc:96] loaded metadata for tablet 37c9ed3e0fed4466bb60613cdc250c9e (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed])
I20260317 21:20:10.513672 10198 tablet_loader.cc:96] loaded metadata for tablet 841735a56c4c4df7a58f93ddbaf67a09 (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019])
I20260317 21:20:10.518216 10198 tablet_loader.cc:96] loaded metadata for tablet ce91b45480ed48a78b76f6936f7a6806 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962])
I20260317 21:20:10.522310 10198 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:20:10.527978 10198 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:20:10.528359 10198 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:20:10.535252 10198 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:20:10.535483 10204 mvcc.cc:204] Tried to move back new op lower bound from 7265412739852152832 to 7265412728360853504. Current Snapshot: MvccSnapshot[applied={T|T < 7265412739852152832 or (T in {7265412739852152832})}]
I20260317 21:20:10.541612 10198 catalog_manager.cc:6055] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Loaded TSK: 0
I20260317 21:20:10.542694 10214 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:10.543279 10198 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:20:10.543442 10214 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:10.548929 10198 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260317 21:20:10.554234 10214 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:10.554560 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:10.557375 10198 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
W20260317 21:20:10.562594 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:10.563089 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:10.564028 10214 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:10.601114 9207 master_service.cc:954] ReplaceTablet: received request to replace tablet 37c9ed3e0fed4466bb60613cdc250c9e from {username='slave'} at 127.0.0.1:44106
I20260317 21:20:10.613658 9203 master_service.cc:954] ReplaceTablet: received request to replace tablet 37c9ed3e0fed4466bb60613cdc250c9e from {username='slave'} at 127.0.0.1:44106
I20260317 21:20:10.599829 9208 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "default.Table_66e3c05d553648b4aab0c59288b3caee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:10.617833 9202 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "default.Table_4bf9ab6d7efc481a9fc1306b6dbd2897" } new_table_name: "default.Table_2e2f41675c9e4d4393bf153766901b13" modify_external_catalogs: true
I20260317 21:20:10.611901 9204 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "default.Table_26888b91e0784e639d5d19ef2ab9e027" } new_table_name: "default.Table_cc4ef5d4d3a54685ae63ef8a6b8f47de" modify_external_catalogs: true
I20260317 21:20:10.619928 9201 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "default.Table_9da3262f4c254404acf6cd7ddac1377c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:10.609537 9205 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6" } modify_external_catalogs: true
I20260317 21:20:10.599187 9209 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "default.Table_8dd898bb0e73488187cc030065d7ced6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:10.633466 9205 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44106:
table { table_name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6" } modify_external_catalogs: true
I20260317 21:20:10.598745 9206 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44106:
name: "default.Table_c25a4ee82851411b8227cf1f4f04844e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260317 21:20:10.649014 9472 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1773782404, eventType=CREATE_TABLE, dbName=default, tableName=table_26888b91e0784e639d5d19ef2ab9e027, message={"server":"","servicePrincipal":"","db":"default","table":"table_26888b91e0784e639d5d19ef2ab9e027","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_26888b91e0784e639d5d19ef2ab9e027\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1773782404},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/warehouse/table_26888b91e0784e639d5d19ef2ab9e027\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_26888b91e0784e639d5d19ef2ab9e027\",\"kudu.table_id\":\"41bfb58e60114fa1a4b52e14d5ff15ed\",\"transient_lastDdlTime\":\"1773782404\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"55c0c553ed67449ba5bba0d7fbc16229\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1773782404,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1773782404, eventType=CREATE_TABLE, dbName=default, tableName=table_9d109e2778144950af7148124af0e79e, message={"server":"","servicePrincipal":"","db":"default","table":"table_9d109e2778144950af7148124af0e79e","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9d109e2778144950af7148124af0e79e\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1773782404},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/warehouse/table_9d109e2778144950af7148124af0e79e\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9d109e2778144950af7148124af0e79e\",\"kudu.table_id\":\"1dc9b1c3c6e847d08861a08accef3b80\",\"transient_lastDdlTime\":\"1773782404\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"55c0c553ed67449ba5bba0d7fbc16229\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1773782404,"files":[]}, messageFormat=json-0.2, catName=hive);
I20260317 21:20:10.651422 9207 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Sending DeleteTablet for 3 replicas of tablet 37c9ed3e0fed4466bb60613cdc250c9e
*** Check failure stack trace: ***
I20260317 21:20:10.659106 9207 catalog_manager.cc:6741] ReplaceTablet: tablet 37c9ed3e0fed4466bb60613cdc250c9e deleted and replaced by tablet 2ffc3b6c364f402bbd9b210902c77e5c
I20260317 21:20:10.689770 9845 tablet_service.cc:1558] Processing DeleteTablet for tablet 37c9ed3e0fed4466bb60613cdc250c9e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ffc3b6c364f402bbd9b210902c77e5c) from {username='slave'} at 127.0.0.1:35910
I20260317 21:20:10.693658 9547 tablet_service.cc:1558] Processing DeleteTablet for tablet 37c9ed3e0fed4466bb60613cdc250c9e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ffc3b6c364f402bbd9b210902c77e5c) from {username='slave'} at 127.0.0.1:44142
I20260317 21:20:10.718151 10222 tablet_replica.cc:333] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:10.718959 10222 raft_consensus.cc:2243] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:10.719692 10222 raft_consensus.cc:2272] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:10.723372 10222 ts_tablet_manager.cc:1916] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:10.729265 9841 tablet_service.cc:1511] Processing CreateTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:10.730420 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ffc3b6c364f402bbd9b210902c77e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:10.732420 9547 tablet_service.cc:1511] Processing CreateTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:10.733873 9547 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ffc3b6c364f402bbd9b210902c77e5c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:10.744792 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet 37c9ed3e0fed4466bb60613cdc250c9e with delete_type TABLET_DATA_DELETED (replaced by tablet 2ffc3b6c364f402bbd9b210902c77e5c) from {username='slave'} at 127.0.0.1:48728
I20260317 21:20:10.746774 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:10.748229 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2ffc3b6c364f402bbd9b210902c77e5c. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f3b5af7b950 google::LogMessage::Flush() at ??:0
@ 0x7f3b5af7b6bf google::LogMessage::~LogMessage() at ??:0
I20260317 21:20:10.805958 10224 tablet_replica.cc:333] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:10.807073 10224 raft_consensus.cc:2243] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:10.807813 10224 raft_consensus.cc:2272] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus is shut down!
@ 0x7f3b6602806c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260317 21:20:10.825577 10221 tablet_replica.cc:333] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:10.826423 10221 raft_consensus.cc:2243] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:20:10.827642 10221 raft_consensus.cc:2272] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:10.843139 10224 ts_tablet_manager.cc:1916] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:10.853583 10221 ts_tablet_manager.cc:1916] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:10.863350 9207 master_service.cc:954] ReplaceTablet: received request to replace tablet 37c9ed3e0fed4466bb60613cdc250c9e from {username='slave'} at 127.0.0.1:44106
I20260317 21:20:10.871650 9207 master_service.cc:954] ReplaceTablet: received request to replace tablet ce91b45480ed48a78b76f6936f7a6806 from {username='slave'} at 127.0.0.1:44106
I20260317 21:20:10.873821 10227 tablet_bootstrap.cc:492] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:10.880225 10227 tablet_bootstrap.cc:654] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:10.885811 10229 tablet_bootstrap.cc:492] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:10.893837 10227 tablet_bootstrap.cc:492] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:10.894276 10227 ts_tablet_manager.cc:1403] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.012s
I20260317 21:20:10.897326 10227 raft_consensus.cc:359] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:10.898053 10227 raft_consensus.cc:385] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:10.899279 10229 tablet_bootstrap.cc:654] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:10.898336 10227 raft_consensus.cc:740] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:10.900127 10227 consensus_queue.cc:260] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:10.902760 10227 ts_tablet_manager.cc:1434] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.008s user 0.007s sys 0.000s
I20260317 21:20:10.906778 10214 raft_consensus.cc:493] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:10.907323 10214 raft_consensus.cc:515] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:10.916949 10222 ts_tablet_manager.cc:1929] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:10.917351 10222 log.cc:1199] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/37c9ed3e0fed4466bb60613cdc250c9e
I20260317 21:20:10.918443 10222 ts_tablet_manager.cc:1950] T 37c9ed3e0fed4466bb60613cdc250c9e P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:10.922115 10214 leader_election.cc:290] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:10.923825 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ffc3b6c364f402bbd9b210902c77e5c" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
W20260317 21:20:10.925302 9798 leader_election.cc:343] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:10.926796 10221 ts_tablet_manager.cc:1929] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:10.927194 10221 log.cc:1199] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/37c9ed3e0fed4466bb60613cdc250c9e
I20260317 21:20:10.927311 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2ffc3b6c364f402bbd9b210902c77e5c" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923" is_pre_election: true
I20260317 21:20:10.928249 10221 ts_tablet_manager.cc:1950] T 37c9ed3e0fed4466bb60613cdc250c9e P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
W20260317 21:20:10.929975 9800 leader_election.cc:343] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Illegal state: must be running to vote when last-logged opid is not known
I20260317 21:20:10.931257 9194 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 37c9ed3e0fed4466bb60613cdc250c9e (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:10.931524 10224 ts_tablet_manager.cc:1929] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:10.930395 9800 leader_election.cc:304] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:10.931932 10224 log.cc:1199] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/37c9ed3e0fed4466bb60613cdc250c9e
I20260317 21:20:10.932883 10224 ts_tablet_manager.cc:1950] T 37c9ed3e0fed4466bb60613cdc250c9e P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:10.932900 10214 raft_consensus.cc:2749] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:10.934340 10229 tablet_bootstrap.cc:492] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:10.934811 10229 ts_tablet_manager.cc:1403] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.049s user 0.012s sys 0.004s
I20260317 21:20:10.936026 9193 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 37c9ed3e0fed4466bb60613cdc250c9e (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:10.941123 10229 raft_consensus.cc:359] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:10.942370 10229 raft_consensus.cc:385] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:10.942775 10229 raft_consensus.cc:740] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:10.945923 9195 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 37c9ed3e0fed4466bb60613cdc250c9e (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:10.951884 10229 consensus_queue.cc:260] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:10.954906 10229 ts_tablet_manager.cc:1434] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.020s user 0.009s sys 0.000s
I20260317 21:20:10.956027 9766 heartbeater.cc:499] Master 127.6.122.126:32989 was elected leader, sending a full tablet report...
I20260317 21:20:10.969849 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet ce91b45480ed48a78b76f6936f7a6806 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e9be2c2f97d465cb1f572e63acf5071) from {username='slave'} at 127.0.0.1:48728
I20260317 21:20:10.970779 10224 tablet_replica.cc:333] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:10.971529 10224 raft_consensus.cc:2243] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:20:10.972707 10224 raft_consensus.cc:2272] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:10.977641 9207 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Sending DeleteTablet for 3 replicas of tablet ce91b45480ed48a78b76f6936f7a6806
I20260317 21:20:10.978374 10224 ts_tablet_manager.cc:1916] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:10.979844 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet ce91b45480ed48a78b76f6936f7a6806 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e9be2c2f97d465cb1f572e63acf5071) from {username='slave'} at 127.0.0.1:44142
I20260317 21:20:10.980551 10222 tablet_replica.cc:333] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:10.981277 10222 raft_consensus.cc:2243] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:10.981806 10222 raft_consensus.cc:2272] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:10.983732 10222 ts_tablet_manager.cc:1916] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:10.990294 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet ce91b45480ed48a78b76f6936f7a6806 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e9be2c2f97d465cb1f572e63acf5071) from {username='slave'} at 127.0.0.1:48728
I20260317 21:20:11.002625 10224 ts_tablet_manager.cc:1929] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:11.003016 10224 log.cc:1199] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/ce91b45480ed48a78b76f6936f7a6806
I20260317 21:20:11.003840 10224 ts_tablet_manager.cc:1950] T ce91b45480ed48a78b76f6936f7a6806 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:11.010378 9195 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet ce91b45480ed48a78b76f6936f7a6806 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
W20260317 21:20:11.011402 9195 catalog_manager.cc:4969] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): delete failed for tablet ce91b45480ed48a78b76f6936f7a6806 because the tablet was not found. No further retry: Not found: Tablet not found: ce91b45480ed48a78b76f6936f7a6806
I20260317 21:20:11.017542 9207 catalog_manager.cc:6741] ReplaceTablet: tablet ce91b45480ed48a78b76f6936f7a6806 deleted and replaced by tablet 5e9be2c2f97d465cb1f572e63acf5071
I20260317 21:20:11.018579 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet ce91b45480ed48a78b76f6936f7a6806 with delete_type TABLET_DATA_DELETED (replaced by tablet 5e9be2c2f97d465cb1f572e63acf5071) from {username='slave'} at 127.0.0.1:35910
I20260317 21:20:11.019510 10221 tablet_replica.cc:333] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:11.020331 10221 raft_consensus.cc:2243] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.020857 10221 raft_consensus.cc:2272] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:11.022780 10221 ts_tablet_manager.cc:1916] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:11.047997 10221 ts_tablet_manager.cc:1929] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:11.048367 10221 log.cc:1199] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/ce91b45480ed48a78b76f6936f7a6806
I20260317 21:20:11.048332 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9be2c2f97d465cb1f572e63acf5071 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:11.049127 10221 ts_tablet_manager.cc:1950] T ce91b45480ed48a78b76f6936f7a6806 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:11.049998 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9be2c2f97d465cb1f572e63acf5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:11.056583 9194 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet ce91b45480ed48a78b76f6936f7a6806 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:11.066762 10231 tablet_bootstrap.cc:492] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:11.069403 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9be2c2f97d465cb1f572e63acf5071 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:11.070413 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9be2c2f97d465cb1f572e63acf5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:11.073534 9841 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9be2c2f97d465cb1f572e63acf5071 (DEFAULT_TABLE table=default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:11.074648 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9be2c2f97d465cb1f572e63acf5071. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:11.080040 9203 master_service.cc:954] ReplaceTablet: received request to replace tablet 2ffc3b6c364f402bbd9b210902c77e5c from {username='slave'} at 127.0.0.1:44106
I20260317 21:20:11.096562 10229 tablet_bootstrap.cc:492] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:11.103022 9203 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Sending DeleteTablet for 3 replicas of tablet 2ffc3b6c364f402bbd9b210902c77e5c
I20260317 21:20:11.103147 10229 tablet_bootstrap.cc:654] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:11.105050 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c with delete_type TABLET_DATA_DELETED (replaced by tablet c40592e077194050b6ed53cf58904059) from {username='slave'} at 127.0.0.1:44142
I20260317 21:20:11.106093 9203 catalog_manager.cc:6741] ReplaceTablet: tablet 2ffc3b6c364f402bbd9b210902c77e5c deleted and replaced by tablet c40592e077194050b6ed53cf58904059
I20260317 21:20:11.106969 9845 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c with delete_type TABLET_DATA_DELETED (replaced by tablet c40592e077194050b6ed53cf58904059) from {username='slave'} at 127.0.0.1:35910
I20260317 21:20:11.107710 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c with delete_type TABLET_DATA_DELETED (replaced by tablet c40592e077194050b6ed53cf58904059) from {username='slave'} at 127.0.0.1:48728
I20260317 21:20:11.108541 10224 tablet_replica.cc:333] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:11.109541 10224 raft_consensus.cc:2243] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.110100 10224 raft_consensus.cc:2272] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:11.112610 10229 tablet_bootstrap.cc:492] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:11.113050 10229 ts_tablet_manager.cc:1403] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.017s user 0.008s sys 0.005s
I20260317 21:20:11.113471 10221 tablet_replica.cc:333] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:11.113700 10224 ts_tablet_manager.cc:1916] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:11.114217 10221 raft_consensus.cc:2243] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.114727 10221 raft_consensus.cc:2272] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:11.115733 10229 raft_consensus.cc:359] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.116474 10229 raft_consensus.cc:385] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.116801 10229 raft_consensus.cc:740] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.117478 10229 consensus_queue.cc:260] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.120131 10229 ts_tablet_manager.cc:1434] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.007s user 0.004s sys 0.002s
I20260317 21:20:11.120499 10221 ts_tablet_manager.cc:1916] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f3b6602782c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260317 21:20:11.135128 10224 ts_tablet_manager.cc:1929] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:11.135596 10224 log.cc:1199] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/2ffc3b6c364f402bbd9b210902c77e5c
I20260317 21:20:11.141019 10231 tablet_bootstrap.cc:654] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:11.145895 10222 ts_tablet_manager.cc:1929] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:11.146355 10222 log.cc:1199] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/ce91b45480ed48a78b76f6936f7a6806
I20260317 21:20:11.147171 10222 ts_tablet_manager.cc:1950] T ce91b45480ed48a78b76f6936f7a6806 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:11.152231 9193 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet ce91b45480ed48a78b76f6936f7a6806 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]) successfully deleted
I20260317 21:20:11.155900 9549 tablet_service.cc:1511] Processing CreateTablet for tablet c40592e077194050b6ed53cf58904059 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:11.157164 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40592e077194050b6ed53cf58904059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:11.160456 9697 tablet_service.cc:1511] Processing CreateTablet for tablet c40592e077194050b6ed53cf58904059 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:11.162040 9697 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40592e077194050b6ed53cf58904059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:11.165529 9845 tablet_service.cc:1511] Processing CreateTablet for tablet c40592e077194050b6ed53cf58904059 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
@ 0x7f3b6602b8a2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260317 21:20:11.166674 10224 ts_tablet_manager.cc:1950] T 2ffc3b6c364f402bbd9b210902c77e5c P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:11.166798 9845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c40592e077194050b6ed53cf58904059. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:11.169162 10227 tablet_bootstrap.cc:492] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:11.170630 10231 tablet_bootstrap.cc:492] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:11.171090 10231 ts_tablet_manager.cc:1403] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.105s user 0.008s sys 0.004s
I20260317 21:20:11.173877 10231 raft_consensus.cc:359] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.174549 10231 raft_consensus.cc:385] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.174798 10231 raft_consensus.cc:740] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.176018 10231 consensus_queue.cc:260] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
W20260317 21:20:11.177568 9193 catalog_manager.cc:4982] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): delete failed for tablet 2ffc3b6c364f402bbd9b210902c77e5c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2ffc3b6c364f402bbd9b210902c77e5c already in progress: creating tablet
I20260317 21:20:11.179823 9195 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 2ffc3b6c364f402bbd9b210902c77e5c (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:11.187330 10227 tablet_bootstrap.cc:654] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:11.198586 10231 ts_tablet_manager.cc:1434] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.027s user 0.012s sys 0.000s
I20260317 21:20:11.199016 9618 heartbeater.cc:499] Master 127.6.122.126:32989 was elected leader, sending a full tablet report...
I20260317 21:20:11.199681 10231 tablet_bootstrap.cc:492] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
@ 0x7f3b6602b85a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260317 21:20:11.207489 10231 tablet_bootstrap.cc:654] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:11.212306 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 2ffc3b6c364f402bbd9b210902c77e5c with delete_type TABLET_DATA_DELETED (replaced by tablet c40592e077194050b6ed53cf58904059) from {username='slave'} at 127.0.0.1:44142
I20260317 21:20:11.213048 10222 tablet_replica.cc:333] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:11.213734 10222 raft_consensus.cc:2243] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.214229 10222 raft_consensus.cc:2272] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:11.216506 10222 ts_tablet_manager.cc:1916] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f3b6602b7ea std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260317 21:20:11.226720 9200 master_service.cc:954] ReplaceTablet: received request to replace tablet 5e9be2c2f97d465cb1f572e63acf5071 from {username='slave'} at 127.0.0.1:44106
I20260317 21:20:11.232432 10229 tablet_bootstrap.cc:492] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:11.235659 10227 tablet_bootstrap.cc:492] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:11.236136 10227 ts_tablet_manager.cc:1403] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.067s user 0.008s sys 0.004s
I20260317 21:20:11.240300 10229 tablet_bootstrap.cc:654] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
@ 0x7f3b6602b7b2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260317 21:20:11.256248 9200 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Sending DeleteTablet for 3 replicas of tablet 5e9be2c2f97d465cb1f572e63acf5071
I20260317 21:20:11.258436 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e9be2c2f97d465cb1f572e63acf5071 with delete_type TABLET_DATA_DELETED (replaced by tablet db8014a841eb44e7be8f1b4737ab2dc6) from {username='slave'} at 127.0.0.1:44142
I20260317 21:20:11.259364 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e9be2c2f97d465cb1f572e63acf5071 with delete_type TABLET_DATA_DELETED (replaced by tablet db8014a841eb44e7be8f1b4737ab2dc6) from {username='slave'} at 127.0.0.1:48728
I20260317 21:20:11.259931 9200 catalog_manager.cc:6741] ReplaceTablet: tablet 5e9be2c2f97d465cb1f572e63acf5071 deleted and replaced by tablet db8014a841eb44e7be8f1b4737ab2dc6
I20260317 21:20:11.260107 10224 tablet_replica.cc:333] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:11.260056 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e9be2c2f97d465cb1f572e63acf5071 with delete_type TABLET_DATA_DELETED (replaced by tablet db8014a841eb44e7be8f1b4737ab2dc6) from {username='slave'} at 127.0.0.1:35910
I20260317 21:20:11.260625 10221 ts_tablet_manager.cc:1929] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:11.260869 10224 raft_consensus.cc:2243] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.261058 10221 log.cc:1199] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/2ffc3b6c364f402bbd9b210902c77e5c
I20260317 21:20:11.261340 10224 raft_consensus.cc:2272] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:11.261767 10221 ts_tablet_manager.cc:1950] T 2ffc3b6c364f402bbd9b210902c77e5c P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:11.263433 10224 ts_tablet_manager.cc:1916] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:11.263330 10227 raft_consensus.cc:359] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.264011 10227 raft_consensus.cc:385] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.264317 10227 raft_consensus.cc:740] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.271085 9194 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 2ffc3b6c364f402bbd9b210902c77e5c (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:11.271813 10227 consensus_queue.cc:260] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.274179 10227 ts_tablet_manager.cc:1434] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.038s user 0.005s sys 0.002s
I20260317 21:20:11.275264 9912 heartbeater.cc:499] Master 127.6.122.126:32989 was elected leader, sending a full tablet report...
I20260317 21:20:11.276795 10227 tablet_bootstrap.cc:492] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:11.277801 10231 tablet_bootstrap.cc:492] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
@ 0x7f3b6602aaae std::__1::__function::__func<>::operator()() at ??:0
I20260317 21:20:11.278261 10231 ts_tablet_manager.cc:1403] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.079s user 0.013s sys 0.013s
@ 0x5ab085 std::__1::__function::__value_func<>::operator()() at ??:?
I20260317 21:20:11.280684 10221 tablet_replica.cc:333] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:11.281481 10221 raft_consensus.cc:2243] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.281942 10221 raft_consensus.cc:2272] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x5aaeba std::__1::function<>::operator()() at ??:?
I20260317 21:20:11.284984 10229 tablet_bootstrap.cc:492] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:11.285447 10229 ts_tablet_manager.cc:1403] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.053s user 0.012s sys 0.000s
I20260317 21:20:11.285192 10231 raft_consensus.cc:359] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.286021 10231 raft_consensus.cc:385] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.286321 10231 raft_consensus.cc:740] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.288470 10231 consensus_queue.cc:260] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.291292 10231 ts_tablet_manager.cc:1434] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.013s user 0.009s sys 0.000s
I20260317 21:20:11.292308 10231 tablet_bootstrap.cc:492] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
W20260317 21:20:11.299005 9200 catalog_manager.cc:5504] Ignoring report from unknown tablet 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:11.299023 10231 tablet_bootstrap.cc:654] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
W20260317 21:20:11.299299 9200 catalog_manager.cc:5504] Ignoring report from unknown tablet 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:11.299485 9200 catalog_manager.cc:5504] Ignoring report from unknown tablet 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:11.305186 10221 ts_tablet_manager.cc:1916] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f3b5d19d74b kudu::Thread::SuperviseThread() at ??:0
@ 0x4749c0 __tsan_thread_start_func at ??:?
I20260317 21:20:11.319806 9203 master_service.cc:954] ReplaceTablet: received request to replace tablet 5e9be2c2f97d465cb1f572e63acf5071 from {username='slave'} at 127.0.0.1:44106
@ 0x7f3b5e8a86db start_thread at ??:0
I20260317 21:20:11.321837 10229 raft_consensus.cc:359] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.329362 10229 raft_consensus.cc:385] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.329645 10229 raft_consensus.cc:740] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.330391 10227 tablet_bootstrap.cc:654] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
@ 0x7f3b59c2061f clone at ??:0
I20260317 21:20:11.345039 10231 tablet_bootstrap.cc:492] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:11.345541 10231 ts_tablet_manager.cc:1403] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.053s user 0.010s sys 0.002s
I20260317 21:20:11.346357 10222 ts_tablet_manager.cc:1929] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:11.346724 10222 log.cc:1199] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/2ffc3b6c364f402bbd9b210902c77e5c
I20260317 21:20:11.347362 10222 ts_tablet_manager.cc:1950] T 2ffc3b6c364f402bbd9b210902c77e5c P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:11.343705 10229 consensus_queue.cc:260] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.349617 10231 raft_consensus.cc:359] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.350307 10231 raft_consensus.cc:385] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.350596 10231 raft_consensus.cc:740] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.351078 10229 ts_tablet_manager.cc:1434] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.065s user 0.007s sys 0.000s
I20260317 21:20:11.351387 10231 consensus_queue.cc:260] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.353665 10231 ts_tablet_manager.cc:1434] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.008s user 0.006s sys 0.001s
I20260317 21:20:11.367849 10227 tablet_bootstrap.cc:492] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:11.368407 10227 ts_tablet_manager.cc:1403] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.092s user 0.015s sys 0.006s
I20260317 21:20:11.369681 10224 ts_tablet_manager.cc:1929] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:11.370106 10224 log.cc:1199] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/5e9be2c2f97d465cb1f572e63acf5071
I20260317 21:20:11.370813 10224 ts_tablet_manager.cc:1950] T 5e9be2c2f97d465cb1f572e63acf5071 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:11.371248 10227 raft_consensus.cc:359] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.371930 10227 raft_consensus.cc:385] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:11.372233 10227 raft_consensus.cc:740] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:11.374159 10222 tablet_replica.cc:333] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:11.375849 10222 raft_consensus.cc:2243] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:11.376327 10222 raft_consensus.cc:2272] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:11.378643 10222 ts_tablet_manager.cc:1916] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:11.381006 10227 consensus_queue.cc:260] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.383836 10221 ts_tablet_manager.cc:1929] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:11.384297 10221 log.cc:1199] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/5e9be2c2f97d465cb1f572e63acf5071
I20260317 21:20:11.385407 10221 ts_tablet_manager.cc:1950] T 5e9be2c2f97d465cb1f572e63acf5071 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:11.390252 10227 ts_tablet_manager.cc:1434] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.022s user 0.007s sys 0.000s
I20260317 21:20:11.396531 10222 ts_tablet_manager.cc:1929] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:11.396957 10222 log.cc:1199] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/5e9be2c2f97d465cb1f572e63acf5071
I20260317 21:20:11.397583 10222 ts_tablet_manager.cc:1950] T 5e9be2c2f97d465cb1f572e63acf5071 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:11.456835 10214 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:11.457338 10214 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:11.459275 10214 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:11.466581 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:11.467478 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:11.467890 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:11.470952 10214 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:11.514453 10239 raft_consensus.cc:493] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:11.514954 10239 raft_consensus.cc:515] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.521880 10239 leader_election.cc:290] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:11.547948 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c40592e077194050b6ed53cf58904059" candidate_uuid: "9b9886349405418f80c5511b013ffbd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923" is_pre_election: true
I20260317 21:20:11.548581 9719 raft_consensus.cc:2468] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9886349405418f80c5511b013ffbd4 in term 0.
I20260317 21:20:11.549990 9503 leader_election.cc:304] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:11.550877 10239 raft_consensus.cc:2804] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:11.551185 10239 raft_consensus.cc:493] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:11.551517 10239 raft_consensus.cc:3060] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:11.559937 10239 raft_consensus.cc:515] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.563267 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c40592e077194050b6ed53cf58904059" candidate_uuid: "9b9886349405418f80c5511b013ffbd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923"
I20260317 21:20:11.563845 9719 raft_consensus.cc:3060] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:11.567579 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c40592e077194050b6ed53cf58904059" candidate_uuid: "9b9886349405418f80c5511b013ffbd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218" is_pre_election: true
I20260317 21:20:11.568245 9865 raft_consensus.cc:2468] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9b9886349405418f80c5511b013ffbd4 in term 0.
I20260317 21:20:11.571611 10239 leader_election.cc:290] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [CANDIDATE]: Term 1 election: Requested vote from peers f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:11.572023 9719 raft_consensus.cc:2468] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9886349405418f80c5511b013ffbd4 in term 1.
I20260317 21:20:11.573122 9865 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c40592e077194050b6ed53cf58904059" candidate_uuid: "9b9886349405418f80c5511b013ffbd4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c2dd7c5a524427da691785d88218218"
I20260317 21:20:11.573712 9865 raft_consensus.cc:3060] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:11.576511 9503 leader_election.cc:304] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:11.577589 10239 raft_consensus.cc:2804] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:20:11.582221 9865 raft_consensus.cc:2468] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9b9886349405418f80c5511b013ffbd4 in term 1.
I20260317 21:20:11.584684 10239 raft_consensus.cc:697] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 1 LEADER]: Becoming Leader. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Running, Role: LEADER
I20260317 21:20:11.585879 10239 consensus_queue.cc:237] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
W20260317 21:20:11.668864 9799 connection.cc:537] server connection from 127.0.0.1:35910 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:11.668803 9920 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:11.669634 9651 connection.cc:537] server connection from 127.0.0.1:48728 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:11.671417 9920 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.122.126:32989)
W20260317 21:20:11.671427 9651 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:11.671415 9798 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:11.672427 9502 connection.cc:537] server connection from 127.0.0.1:44142 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:11.677634 9920 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.126:32989)
W20260317 21:20:11.678689 9920 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.122.126:32989)
W20260317 21:20:11.681344 9920 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.122.126:32989)
I20260317 21:20:11.821758 10213 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:11.822264 10213 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:11.824100 10213 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:11.826925 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:20:11.827807 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:11.828225 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:11.829854 10213 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:11.838299 10262 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:11.839396 10262 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:11.843542 10262 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:11.859186 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:11.868149 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:11.875703 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:11.876309 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:11.880173 10262 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 21:20:12.000882 10267 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:12.001507 10267 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:12.003304 10267 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:12.005308 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:12.005879 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:12.006301 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:12.007091 10267 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:12.379528 10268 consensus_queue.cc:1048] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:20:12.406735 10272 consensus_queue.cc:1048] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:20:12.587800 10279 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:12.588184 10279 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:12.590345 10279 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:12.599339 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:12.599885 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:12.600373 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:12.601128 10279 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 21:20:13.017767 6633 external_mini_cluster.cc:1366] Running /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
/tmp/dist-test-tasklBwffU/build/tsan/bin/kudu
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.126:32989
--webserver_interface=127.6.122.126
--webserver_port=43267
--builtin_ntp_servers=127.6.122.84:45889
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--hive_metastore_uris=thrift://127.0.0.1:42187
--hive_metastore_notification_log_poll_period_seconds=1
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--catalog_manager_bg_task_wait_ms=10
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20260317 21:20:13.349557 10291 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:13.350009 10291 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:13.356220 10291 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:13.362495 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:13.363983 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:20:13.365392 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:13.366189 10291 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260317 21:20:13.393760 10286 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260317 21:20:13.394534 10286 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260317 21:20:13.395195 10286 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260317 21:20:13.454340 10286 flags.cc:425] Enabled experimental flag: --enable_metadata_cleanup_for_deleted_tables_and_tablets=true
W20260317 21:20:13.454687 10286 flags.cc:425] Enabled experimental flag: --metadata_for_deleted_table_and_tablet_reserved_secs=0
W20260317 21:20:13.454969 10286 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260317 21:20:13.455233 10286 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260317 21:20:13.455494 10286 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260317 21:20:13.455749 10286 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260317 21:20:13.501892 10286 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.6.122.84:45889
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
--hive_metastore_uris=thrift://127.0.0.1:42187
--catalog_manager_bg_task_wait_ms=10
--enable_metadata_cleanup_for_deleted_tables_and_tablets=true
--metadata_for_deleted_table_and_tablet_reserved_secs=0
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.6.122.126:32989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.6.122.126
--webserver_port=43267
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 07148adc8b1ea21ab089857e39de46f7fa68ddab
build type FASTDEBUG
built by None at 17 Mar 2026 21:07:52 UTC on e7f111948823
build id 10979
TSAN enabled
I20260317 21:20:13.504586 10286 env_posix.cc:2267] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260317 21:20:13.506645 10286 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20260317 21:20:13.520095 10301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:14.590273 10314 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:14.591836 10314 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:14.596040 10314 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:14.639741 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:14.641517 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:20:14.642222 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:14.643384 10314 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 21:20:14.670888 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:14.671546 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:20:13.522378 10302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260317 21:20:14.927299 10300 debug-util.cc:398] Leaking SignalData structure 0x7b0800037b00 after lost signal to thread 10286
W20260317 21:20:15.027099 10286 thread.cc:641] GCE (cloud detector) Time spent creating pthread: real 1.508s user 0.536s sys 0.968s
W20260317 21:20:15.027498 10286 thread.cc:608] GCE (cloud detector) Time spent starting thread: real 1.509s user 0.536s sys 0.968s
W20260317 21:20:15.036147 10319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260317 21:20:15.036584 10286 server_base.cc:1060] running on GCE node
I20260317 21:20:15.037806 10286 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260317 21:20:15.040243 10286 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260317 21:20:15.041649 10286 hybrid_clock.cc:648] HybridClock initialized: now 1773782415041610 us; error 35 us; skew 500 ppm
I20260317 21:20:15.042552 10286 server_base.cc:860] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260317 21:20:15.048936 10286 webserver.cc:492] Webserver started at http://127.6.122.126:43267/ using document root <none> and password file <none>
I20260317 21:20:15.049953 10286 fs_manager.cc:362] Metadata directory not provided
I20260317 21:20:15.050249 10286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260317 21:20:15.059597 10286 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.005s
I20260317 21:20:15.064949 10326 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:15.065387 10326 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:15.066140 10325 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260317 21:20:15.067389 10326 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:15.067593 10286 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.005s
W20260317 21:20:15.068634 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:15.069169 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:15.069609 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:15.070369 10326 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:15.068043 10286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data,/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
uuid: "d6b9687775954282a11f2cf829d24bac"
format_stamp: "Formatted at 2026-03-17 21:19:50 on dist-test-slave-979h"
I20260317 21:20:15.071282 10286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260317 21:20:15.132396 10286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260317 21:20:15.134050 10286 env_posix.cc:2267] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260317 21:20:15.134581 10286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260317 21:20:15.175194 10349 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:15.175777 10349 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:15.177644 10349 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:15.183122 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:15.184855 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:15.185352 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:15.186368 10349 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260317 21:20:15.222718 10286 rpc_server.cc:307] RPC server started. Bound to: 127.6.122.126:32989
I20260317 21:20:15.222796 10382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.6.122.126:32989 every 8 connection(s)
I20260317 21:20:15.226423 10286 server_base.cc:1192] Dumped server information to /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/master-0/data/info.pb
I20260317 21:20:15.230893 6633 external_mini_cluster.cc:1428] Started /tmp/dist-test-tasklBwffU/build/tsan/bin/kudu as pid 10286
I20260317 21:20:15.247113 10384 sys_catalog.cc:263] Verifying existing consensus state
I20260317 21:20:15.255338 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Bootstrap starting.
I20260317 21:20:15.292275 10384 log.cc:826] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Log is configured to *not* fsync() on all Append() calls
I20260317 21:20:15.407583 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Bootstrap replayed 1/1 log segments. Stats: ops{read=45 overwritten=0 applied=45 ignored=0} inserts{seen=24 ignored=0} mutations{seen=46 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260317 21:20:15.408610 10384 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Bootstrap complete.
I20260317 21:20:15.427570 10384 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:15.430130 10384 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6b9687775954282a11f2cf829d24bac, State: Initialized, Role: FOLLOWER
I20260317 21:20:15.430930 10384 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 45, Last appended: 3.45, Last appended by leader: 45, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:15.433652 10389 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:15.434556 10389 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:15.435361 10384 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: configured and running, proceeding with master startup.
I20260317 21:20:15.547148 9912 heartbeater.cc:344] Connected to a master server at 127.6.122.126:32989
I20260317 21:20:15.557968 10344 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "6c2dd7c5a524427da691785d88218218" instance_seqno: 1773782403703098) as {username='slave'} at 127.6.122.67:59691; Asking this server to re-register.
I20260317 21:20:15.559625 9766 heartbeater.cc:344] Connected to a master server at 127.6.122.126:32989
I20260317 21:20:15.563968 9912 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:15.564769 10403 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:20:15.565111 10403 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: loading cluster ID for follower catalog manager: success
I20260317 21:20:15.570113 10342 ts_manager.cc:194] Registered new tserver with Master: 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
I20260317 21:20:15.570359 10343 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" instance_seqno: 1773782401247982) as {username='slave'} at 127.6.122.66:37123; Asking this server to re-register.
I20260317 21:20:15.572822 9766 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:15.578722 10403 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: acquiring CA information for follower catalog manager: success
I20260317 21:20:15.579344 10344 ts_manager.cc:194] Registered new tserver with Master: f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
I20260317 21:20:15.586930 10403 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260317 21:20:15.574672 10286 master_runner.cc:186] Error getting master registration for 127.6.122.125:38261: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111),
I20260317 21:20:15.597757 10286 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:15.638829 9618 heartbeater.cc:344] Connected to a master server at 127.6.122.126:32989
I20260317 21:20:15.642856 10344 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "9b9886349405418f80c5511b013ffbd4" instance_seqno: 1773782398616276) as {username='slave'} at 127.6.122.65:42557; Asking this server to re-register.
I20260317 21:20:15.644559 9618 heartbeater.cc:461] Registering TS with master...
I20260317 21:20:15.648133 10344 ts_manager.cc:194] Registered new tserver with Master: 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:20:15.941941 10410 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:15.942474 10410 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:15.945391 10410 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
W20260317 21:20:15.951901 10329 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:15.957602 10329 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:15.959329 10135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" is_pre_election: true
I20260317 21:20:15.959987 10135 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6b9687775954282a11f2cf829d24bac in term 3.
I20260317 21:20:15.961377 10329 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d
I20260317 21:20:15.962159 10410 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260317 21:20:15.962532 10410 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:15.962838 10410 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 3 FOLLOWER]: Advancing to term 4
I20260317 21:20:15.969741 10410 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:15.971663 10410 leader_election.cc:290] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 4 election: Requested vote from peers a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261), ed19cc1cc9e74a0d9b8dc4a609f1bea9 (127.6.122.124:41015)
I20260317 21:20:15.974121 10135 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6b9687775954282a11f2cf829d24bac" candidate_term: 4 candidate_status { last_received { term: 3 index: 45 } } ignore_live_leader: false dest_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9"
I20260317 21:20:15.974733 10135 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 3 FOLLOWER]: Advancing to term 4
W20260317 21:20:15.975908 10329 leader_election.cc:336] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:15.982084 10135 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6b9687775954282a11f2cf829d24bac in term 4.
I20260317 21:20:15.984086 10329 leader_election.cc:304] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d6b9687775954282a11f2cf829d24bac, ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d
I20260317 21:20:15.985018 10410 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 4 FOLLOWER]: Leader election won for term 4
I20260317 21:20:15.986967 10410 raft_consensus.cc:697] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [term 4 LEADER]: Becoming Leader. State: Replica: d6b9687775954282a11f2cf829d24bac, State: Running, Role: LEADER
I20260317 21:20:15.988256 10410 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 45, Committed index: 45, Last appended: 3.45, Last appended by leader: 45, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:15.998493 10414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b9687775954282a11f2cf829d24bac. Latest consensus state: current_term: 4 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:15.999365 10414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: LEADER
I20260317 21:20:16.002014 10416 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20260317 21:20:16.010231 10416 catalog_manager.cc:679] Loaded metadata for table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]
I20260317 21:20:16.011775 10416 catalog_manager.cc:679] Loaded metadata for table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962]
I20260317 21:20:16.014106 10416 catalog_manager.cc:679] Loaded metadata for table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019]
I20260317 21:20:16.022979 10416 tablet_loader.cc:96] loaded metadata for tablet 841735a56c4c4df7a58f93ddbaf67a09 (table default.table_4bf9ab6d7efc481a9fc1306b6dbd2897 [id=b14c029174b947aba867883bd0fd8019])
I20260317 21:20:16.024216 10416 tablet_loader.cc:96] loaded metadata for tablet c40592e077194050b6ed53cf58904059 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed])
I20260317 21:20:16.025642 10416 tablet_loader.cc:96] loaded metadata for tablet db8014a841eb44e7be8f1b4737ab2dc6 (table default.table_86ffaaa1f6af420b83eff965c1f2b4c6 [id=468218bf40694b02b7d92fc4da2d8962])
I20260317 21:20:16.027190 10416 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20260317 21:20:16.032509 10416 catalog_manager.cc:1269] Loaded cluster ID: 55c0c553ed67449ba5bba0d7fbc16229
I20260317 21:20:16.032934 10416 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20260317 21:20:16.038641 10416 catalog_manager.cc:1514] Loading token signing keys...
I20260317 21:20:16.043434 10416 catalog_manager.cc:6055] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Loaded TSK: 0
I20260317 21:20:16.045001 10416 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20260317 21:20:16.049573 10416 catalog_manager.cc:1537] Loading latest processed Hive Metastore notification log event ID...
I20260317 21:20:16.055193 10416 catalog_manager.cc:5893] Last processed Hive Metastore notification event ID: -1
I20260317 21:20:16.076457 10135 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Refusing update from remote peer d6b9687775954282a11f2cf829d24bac: Log matching property violated. Preceding OpId in replica: term: 3 index: 45. Preceding OpId from leader: term: 4 index: 47. (index mismatch)
I20260317 21:20:16.078941 10414 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [LEADER]: Connected to new peer: Peer: permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 46, Last known committed idx: 45, Time since last communication: 0.000s
W20260317 21:20:16.083271 10329 consensus_peers.cc:597] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac -> Peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Couldn't send request to peer a4719f316c614a9cadaf5edab834804d. Status: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260317 21:20:16.084523 10343 master_service.cc:954] ReplaceTablet: received request to replace tablet c40592e077194050b6ed53cf58904059 from {username='slave'} at 127.0.0.1:55560
I20260317 21:20:16.090564 10341 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55560:
table { table_name: "default.Table_26888b91e0784e639d5d19ef2ab9e027" } new_table_name: "default.Table_cc4ef5d4d3a54685ae63ef8a6b8f47de" modify_external_catalogs: true
I20260317 21:20:16.091343 10339 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55560:
table { table_name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6" } modify_external_catalogs: true
I20260317 21:20:16.092350 10339 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55560:
table { table_name: "default.Table_86ffaaa1f6af420b83eff965c1f2b4c6" } modify_external_catalogs: true
I20260317 21:20:16.093556 10340 master_service.cc:954] ReplaceTablet: received request to replace tablet c40592e077194050b6ed53cf58904059 from {username='slave'} at 127.0.0.1:55560
I20260317 21:20:16.095254 10342 catalog_manager.cc:3544] Servicing AlterTable request from {username='slave'} at 127.0.0.1:55560:
table { table_name: "default.Table_4bf9ab6d7efc481a9fc1306b6dbd2897" } new_table_name: "default.Table_2e2f41675c9e4d4393bf153766901b13" modify_external_catalogs: true
I20260317 21:20:16.098606 10338 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55560:
name: "default.Table_8dd898bb0e73488187cc030065d7ced6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:16.102030 10337 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55560:
name: "default.Table_c25a4ee82851411b8227cf1f4f04844e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:16.102887 10336 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55560:
name: "default.Table_66e3c05d553648b4aab0c59288b3caee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
F20260317 21:20:16.104705 10402 hms_notification_log_listener.cc:287] non-monotonous event IDs from HMS: current 1, previous 1; dumping first 2 out of 4 received events: NotificationEvent(eventId=1, eventTime=1773782404, eventType=CREATE_TABLE, dbName=default, tableName=table_26888b91e0784e639d5d19ef2ab9e027, message={"server":"","servicePrincipal":"","db":"default","table":"table_26888b91e0784e639d5d19ef2ab9e027","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_26888b91e0784e639d5d19ef2ab9e027\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1773782404},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/warehouse/table_26888b91e0784e639d5d19ef2ab9e027\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_26888b91e0784e639d5d19ef2ab9e027\",\"kudu.table_id\":\"41bfb58e60114fa1a4b52e14d5ff15ed\",\"transient_lastDdlTime\":\"1773782404\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"55c0c553ed67449ba5bba0d7fbc16229\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1773782404,"files":[]}, messageFormat=json-0.2, catName=hive); NotificationEvent(eventId=1, eventTime=1773782404, eventType=CREATE_TABLE, dbName=default, tableName=table_9d109e2778144950af7148124af0e79e, message={"server":"","servicePrincipal":"","db":"default","table":"table_9d109e2778144950af7148124af0e79e","tableType":"MANAGED_TABLE","tableObjJson":"{\"1\":{\"str\":\"table_9d109e2778144950af7148124af0e79e\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"slave\"},\"4\":{\"i32\":1773782404},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",1,{\"1\":{\"str\":\"key\"},\"2\":{\"str\":\"int\"},\"3\":{\"str\":\"\"}}]},\"2\":{\"str\":\"file:/tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/warehouse/table_9d109e2778144950af7148124af0e79e\"},\"3\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"str\":\"\"},\"2\":{\"str\":\"org.apache.hadoop.hive.kudu.KuduSerDe\"},\"3\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",7,{\"kudu.table_name\":\"default.table_9d109e2778144950af7148124af0e79e\",\"kudu.table_id\":\"1dc9b1c3c6e847d08861a08accef3b80\",\"transient_lastDdlTime\":\"1773782404\",\"comment\":\"\",\"kudu.master_addresses\":\"127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015\",\"storage_handler\":\"org.apache.hadoop.hive.kudu.KuduStorageHandler\",\"kudu.cluster_id\":\"55c0c553ed67449ba5bba0d7fbc16229\"}]},\"10\":{\"str\":\"\"},\"11\":{\"str\":\"\"},\"12\":{\"str\":\"MANAGED_TABLE\"},\"14\":{\"tf\":0},\"17\":{\"str\":\"hive\"},\"18\":{\"i32\":1}}","timestamp":1773782404,"files":[]}, messageFormat=json-0.2, catName=hive);
*** Check failure stack trace: ***
I20260317 21:20:16.108619 10344 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55560:
name: "default.Table_9da3262f4c254404acf6cd7ddac1377c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260317 21:20:16.121712 10418 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6b9687775954282a11f2cf829d24bac. Latest consensus state: current_term: 4 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:16.122509 10418 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:16.131114 10417 mvcc.cc:204] Tried to move back new op lower bound from 7265412776221970432 to 7265412775900782592. Current Snapshot: MvccSnapshot[applied={T|T < 7265412776221970432}]
@ 0x7f867a324950 google::LogMessage::Flush() at ??:0
I20260317 21:20:16.137948 10414 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:16.138744 10414 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac [sys.catalog]: This master's current role is: LEADER
I20260317 21:20:16.153055 10418 sys_catalog.cc:455] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "d6b9687775954282a11f2cf829d24bac" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } } }
I20260317 21:20:16.153832 10418 sys_catalog.cc:458] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [sys.catalog]: This master's current role is: FOLLOWER
I20260317 21:20:16.159765 10343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Sending DeleteTablet for 3 replicas of tablet c40592e077194050b6ed53cf58904059
I20260317 21:20:16.168241 10343 catalog_manager.cc:6741] ReplaceTablet: tablet c40592e077194050b6ed53cf58904059 deleted and replaced by tablet 5d0295fa6599447f8712ed20ae05a4e3
I20260317 21:20:16.175485 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet c40592e077194050b6ed53cf58904059 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0295fa6599447f8712ed20ae05a4e3) from {username='slave'} at 127.0.0.1:46624
I20260317 21:20:16.177526 10425 tablet_replica.cc:333] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:16.178265 10425 raft_consensus.cc:2243] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 1 LEADER]: Raft consensus shutting down.
I20260317 21:20:16.179184 10425 raft_consensus.cc:2272] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:16.180755 10425 ts_tablet_manager.cc:1916] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f867a3246bf google::LogMessage::~LogMessage() at ??:0
I20260317 21:20:16.205241 9549 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:16.206324 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0295fa6599447f8712ed20ae05a4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:16.209563 10425 ts_tablet_manager.cc:1929] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:16.223801 9845 tablet_service.cc:1558] Processing DeleteTablet for tablet c40592e077194050b6ed53cf58904059 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0295fa6599447f8712ed20ae05a4e3) from {username='slave'} at 127.0.0.1:53520
I20260317 21:20:16.225708 9841 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:16.227113 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0295fa6599447f8712ed20ae05a4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:16.228037 10429 tablet_replica.cc:333] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:16.228924 10429 raft_consensus.cc:2243] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:16.229558 10429 raft_consensus.cc:2272] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:16.229976 9697 tablet_service.cc:1558] Processing DeleteTablet for tablet c40592e077194050b6ed53cf58904059 with delete_type TABLET_DATA_DELETED (replaced by tablet 5d0295fa6599447f8712ed20ae05a4e3) from {username='slave'} at 127.0.0.1:35044
I20260317 21:20:16.231360 9698 tablet_service.cc:1511] Processing CreateTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:16.232383 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d0295fa6599447f8712ed20ae05a4e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:16.233009 10429 ts_tablet_manager.cc:1916] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f86853d106c kudu::master::HmsNotificationLogListenerTask::Poll() at ??:0
I20260317 21:20:16.237006 10430 tablet_replica.cc:333] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:16.245532 10430 raft_consensus.cc:2243] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:16.246306 10430 raft_consensus.cc:2272] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:16.248929 10430 ts_tablet_manager.cc:1916] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:16.209977 10425 log.cc:1199] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/c40592e077194050b6ed53cf58904059
I20260317 21:20:16.251374 10425 ts_tablet_manager.cc:1950] T c40592e077194050b6ed53cf58904059 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:16.265605 10327 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet c40592e077194050b6ed53cf58904059 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:16.266172 10431 tablet_bootstrap.cc:492] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:16.279641 10431 tablet_bootstrap.cc:654] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:16.288234 10429 ts_tablet_manager.cc:1929] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260317 21:20:16.288822 10429 log.cc:1199] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/c40592e077194050b6ed53cf58904059
I20260317 21:20:16.289834 10429 ts_tablet_manager.cc:1950] T c40592e077194050b6ed53cf58904059 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:16.291841 10434 tablet_bootstrap.cc:492] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
I20260317 21:20:16.295703 10433 tablet_bootstrap.cc:492] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
I20260317 21:20:16.296861 10328 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet c40592e077194050b6ed53cf58904059 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:16.301885 10430 ts_tablet_manager.cc:1929] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
@ 0x7f86853d082c kudu::master::HmsNotificationLogListenerTask::RunLoop() at ??:0
I20260317 21:20:16.305352 10431 tablet_bootstrap.cc:492] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:16.305806 10431 ts_tablet_manager.cc:1403] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.040s user 0.015s sys 0.000s
I20260317 21:20:16.308444 10431 raft_consensus.cc:359] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:16.309187 10431 raft_consensus.cc:385] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:16.309477 10431 raft_consensus.cc:740] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:16.310115 10431 consensus_queue.cc:260] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:16.311548 10430 log.cc:1199] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/c40592e077194050b6ed53cf58904059
I20260317 21:20:16.312434 10430 ts_tablet_manager.cc:1950] T c40592e077194050b6ed53cf58904059 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:16.313642 10433 tablet_bootstrap.cc:654] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:16.315886 10329 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet c40592e077194050b6ed53cf58904059 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
@ 0x7f86853d48a2 kudu::master::HmsNotificationLogListenerTask::Init()::$_0::operator()() at ??:0
I20260317 21:20:16.327329 10434 tablet_bootstrap.cc:654] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:16.334619 10431 ts_tablet_manager.cc:1434] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.029s user 0.007s sys 0.016s
I20260317 21:20:16.337054 9618 heartbeater.cc:499] Master 127.6.122.126:32989 was elected leader, sending a full tablet report...
@ 0x7f86853d485a _ZNSt3__18__invokeIRZN4kudu6master30HmsNotificationLogListenerTask4InitEvE3$_0JEEEDTclclsr3std3__1E7forwardIT_Efp_Espclsr3std3__1E7forwardIT0_Efp0_EEEOS6_DpOS7_ at ??:0
I20260317 21:20:16.352049 10434 tablet_bootstrap.cc:492] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:16.352548 10434 ts_tablet_manager.cc:1403] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.061s user 0.006s sys 0.008s
I20260317 21:20:16.362828 10433 tablet_bootstrap.cc:492] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:16.363307 10433 ts_tablet_manager.cc:1403] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.068s user 0.009s sys 0.003s
I20260317 21:20:16.366307 10433 raft_consensus.cc:359] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:16.367367 10433 raft_consensus.cc:385] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:16.367698 10433 raft_consensus.cc:740] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:16.368602 10433 consensus_queue.cc:260] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
@ 0x7f86853d47ea std::__1::__invoke_void_return_wrapper<>::__call<>() at ??:0
I20260317 21:20:16.372697 10343 master_service.cc:954] ReplaceTablet: received request to replace tablet 5d0295fa6599447f8712ed20ae05a4e3 from {username='slave'} at 127.0.0.1:55560
I20260317 21:20:16.376108 10433 ts_tablet_manager.cc:1434] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.012s user 0.009s sys 0.000s
I20260317 21:20:16.372105 10434 raft_consensus.cc:359] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:16.376963 10434 raft_consensus.cc:385] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:16.377310 10434 raft_consensus.cc:740] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:16.392241 10340 master_service.cc:954] ReplaceTablet: received request to replace tablet 5d0295fa6599447f8712ed20ae05a4e3 from {username='slave'} at 127.0.0.1:55560
I20260317 21:20:16.397470 9766 heartbeater.cc:499] Master 127.6.122.126:32989 was elected leader, sending a full tablet report...
I20260317 21:20:16.398869 10343 catalog_manager.cc:5958] T 00000000000000000000000000000000 P d6b9687775954282a11f2cf829d24bac: Sending DeleteTablet for 3 replicas of tablet 5d0295fa6599447f8712ed20ae05a4e3
I20260317 21:20:16.400473 10343 catalog_manager.cc:6741] ReplaceTablet: tablet 5d0295fa6599447f8712ed20ae05a4e3 deleted and replaced by tablet b92c2321df1b48fb93cb41d311ff2560
I20260317 21:20:16.400988 9549 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 with delete_type TABLET_DATA_DELETED (replaced by tablet b92c2321df1b48fb93cb41d311ff2560) from {username='slave'} at 127.0.0.1:46624
I20260317 21:20:16.402240 10425 tablet_replica.cc:333] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: stopping tablet replica
I20260317 21:20:16.402907 10425 raft_consensus.cc:2243] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:16.403374 10425 raft_consensus.cc:2272] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260317 21:20:16.403568 9698 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 with delete_type TABLET_DATA_DELETED (replaced by tablet b92c2321df1b48fb93cb41d311ff2560) from {username='slave'} at 127.0.0.1:35044
I20260317 21:20:16.404145 9841 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 with delete_type TABLET_DATA_DELETED (replaced by tablet b92c2321df1b48fb93cb41d311ff2560) from {username='slave'} at 127.0.0.1:53520
I20260317 21:20:16.405522 10430 tablet_replica.cc:333] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: stopping tablet replica
I20260317 21:20:16.406231 10430 raft_consensus.cc:2243] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:16.406697 10430 raft_consensus.cc:2272] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Raft consensus is shut down!
W20260317 21:20:16.410064 10328 catalog_manager.cc:4982] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): delete failed for tablet 5d0295fa6599447f8712ed20ae05a4e3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5d0295fa6599447f8712ed20ae05a4e3 already in progress: creating tablet
I20260317 21:20:16.412681 10425 ts_tablet_manager.cc:1916] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x7f86853d47b2 std::__1::__function::__alloc_func<>::operator()() at ??:0
I20260317 21:20:16.409185 10434 consensus_queue.cc:260] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:16.423740 10430 ts_tablet_manager.cc:1916] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260317 21:20:16.453588 9841 tablet_service.cc:1511] Processing CreateTablet for tablet b92c2321df1b48fb93cb41d311ff2560 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:16.453940 9912 heartbeater.cc:499] Master 127.6.122.126:32989 was elected leader, sending a full tablet report...
I20260317 21:20:16.455641 10434 ts_tablet_manager.cc:1434] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.103s user 0.007s sys 0.008s
I20260317 21:20:16.459698 9841 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b92c2321df1b48fb93cb41d311ff2560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:16.461793 10430 ts_tablet_manager.cc:1929] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:16.462162 10430 log.cc:1199] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-1/wal/wals/5d0295fa6599447f8712ed20ae05a4e3
I20260317 21:20:16.462823 10430 ts_tablet_manager.cc:1950] T 5d0295fa6599447f8712ed20ae05a4e3 P f738662691924b8a8f9b4b7cf1804923: Deleting consensus metadata
I20260317 21:20:16.470688 10329 catalog_manager.cc:5002] TS f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): tablet 5d0295fa6599447f8712ed20ae05a4e3 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
W20260317 21:20:16.475625 10335 catalog_manager.cc:5504] Ignoring report from unknown tablet 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:16.475979 10335 catalog_manager.cc:5504] Ignoring report from unknown tablet 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:16.476189 10335 catalog_manager.cc:5504] Ignoring report from unknown tablet 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:16.480535 9845 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d0295fa6599447f8712ed20ae05a4e3 with delete_type TABLET_DATA_DELETED (replaced by tablet b92c2321df1b48fb93cb41d311ff2560) from {username='slave'} at 127.0.0.1:53520
I20260317 21:20:16.481573 10429 tablet_replica.cc:333] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: stopping tablet replica
I20260317 21:20:16.482638 10429 raft_consensus.cc:2243] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260317 21:20:16.483228 10429 raft_consensus.cc:2272] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Raft consensus is shut down!
@ 0x7f86853d3aae std::__1::__function::__func<>::operator()() at ??:0
@ 0x5ab085 std::__1::__function::__value_func<>::operator()() at ??:?
I20260317 21:20:16.486899 10425 ts_tablet_manager.cc:1929] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:16.487314 10425 log.cc:1199] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-0/wal/wals/5d0295fa6599447f8712ed20ae05a4e3
I20260317 21:20:16.488037 10425 ts_tablet_manager.cc:1950] T 5d0295fa6599447f8712ed20ae05a4e3 P 9b9886349405418f80c5511b013ffbd4: Deleting consensus metadata
I20260317 21:20:16.487931 10429 ts_tablet_manager.cc:1916] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Deleting tablet data with delete state TABLET_DATA_DELETED
@ 0x5aaeba std::__1::function<>::operator()() at ??:?
I20260317 21:20:16.503938 9698 tablet_service.cc:1511] Processing CreateTablet for tablet b92c2321df1b48fb93cb41d311ff2560 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:16.505106 9698 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b92c2321df1b48fb93cb41d311ff2560. 1 dirs total, 0 dirs full, 0 dirs failed
I20260317 21:20:16.528556 9549 tablet_service.cc:1511] Processing CreateTablet for tablet b92c2321df1b48fb93cb41d311ff2560 (DEFAULT_TABLE table=default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]), partition=RANGE (key) PARTITION UNBOUNDED
I20260317 21:20:16.530269 9549 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b92c2321df1b48fb93cb41d311ff2560. 1 dirs total, 0 dirs full, 0 dirs failed
@ 0x7f867c54674b kudu::Thread::SuperviseThread() at ??:0
@ 0x4749c0 __tsan_thread_start_func at ??:?
I20260317 21:20:16.525857 10327 catalog_manager.cc:5002] TS 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): tablet 5d0295fa6599447f8712ed20ae05a4e3 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:16.570688 10433 tablet_bootstrap.cc:492] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923: Bootstrap starting.
@ 0x7f867dc516db start_thread at ??:0
I20260317 21:20:16.574498 10429 ts_tablet_manager.cc:1929] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260317 21:20:16.574954 10429 log.cc:1199] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Deleting WAL directory at /tmp/dist-test-tasklBwffU/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_3.1773782310190621-6633-0/minicluster-data/ts-2/wal/wals/5d0295fa6599447f8712ed20ae05a4e3
I20260317 21:20:16.575805 10429 ts_tablet_manager.cc:1950] T 5d0295fa6599447f8712ed20ae05a4e3 P 6c2dd7c5a524427da691785d88218218: Deleting consensus metadata
I20260317 21:20:16.578416 10431 tablet_bootstrap.cc:492] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4: Bootstrap starting.
I20260317 21:20:16.578599 10328 catalog_manager.cc:5002] TS 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): tablet 5d0295fa6599447f8712ed20ae05a4e3 (table default.table_26888b91e0784e639d5d19ef2ab9e027 [id=41bfb58e60114fa1a4b52e14d5ff15ed]) successfully deleted
I20260317 21:20:16.578953 10433 tablet_bootstrap.cc:654] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:16.586988 10431 tablet_bootstrap.cc:654] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:16.590464 10434 tablet_bootstrap.cc:492] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218: Bootstrap starting.
@ 0x7f8678fc961f clone at ??:0
I20260317 21:20:16.598448 10433 tablet_bootstrap.cc:492] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923: No bootstrap required, opened a new log
I20260317 21:20:16.598908 10433 ts_tablet_manager.cc:1403] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.010s
I20260317 21:20:16.598968 10434 tablet_bootstrap.cc:654] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218: Neither blocks nor log segments found. Creating new log.
I20260317 21:20:16.602375 10433 raft_consensus.cc:359] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.603295 10433 raft_consensus.cc:385] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:16.603567 10433 raft_consensus.cc:740] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f738662691924b8a8f9b4b7cf1804923, State: Initialized, Role: FOLLOWER
I20260317 21:20:16.605897 10433 consensus_queue.cc:260] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.609344 10434 tablet_bootstrap.cc:492] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218: No bootstrap required, opened a new log
I20260317 21:20:16.609824 10434 ts_tablet_manager.cc:1403] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.011s
I20260317 21:20:16.616854 10433 ts_tablet_manager.cc:1434] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923: Time spent starting tablet: real 0.018s user 0.008s sys 0.003s
I20260317 21:20:16.619096 10434 raft_consensus.cc:359] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.621470 10431 tablet_bootstrap.cc:492] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4: No bootstrap required, opened a new log
I20260317 21:20:16.621390 10434 raft_consensus.cc:385] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:16.621716 10434 raft_consensus.cc:740] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Initialized, Role: FOLLOWER
I20260317 21:20:16.621898 10431 ts_tablet_manager.cc:1403] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4: Time spent bootstrapping tablet: real 0.044s user 0.020s sys 0.007s
I20260317 21:20:16.622407 10434 consensus_queue.cc:260] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.625104 10434 ts_tablet_manager.cc:1434] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218: Time spent starting tablet: real 0.008s user 0.002s sys 0.004s
I20260317 21:20:16.625423 10431 raft_consensus.cc:359] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.626142 10431 raft_consensus.cc:385] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260317 21:20:16.626332 10431 raft_consensus.cc:740] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9b9886349405418f80c5511b013ffbd4, State: Initialized, Role: FOLLOWER
I20260317 21:20:16.626937 10431 consensus_queue.cc:260] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.633926 10431 ts_tablet_manager.cc:1434] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4: Time spent starting tablet: real 0.012s user 0.007s sys 0.000s
W20260317 21:20:16.846119 9651 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:16.846130 9798 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:16.846199 9501 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:16.848289 9920 connection.cc:537] client connection to 127.6.122.126:32989 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260317 21:20:16.850042 9920 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.6.122.126:32989)
W20260317 21:20:16.850754 9920 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.6.122.126:32989)
W20260317 21:20:16.852298 9920 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.6.122.126:32989)
W20260317 21:20:16.853096 9920 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.6.122.126:32989)
I20260317 21:20:16.937263 10457 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:16.938452 10457 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:16.940888 10457 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:16.944514 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260317 21:20:16.949092 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:16.949744 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:16.950161 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:16.950896 10457 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:16.994055 10444 raft_consensus.cc:493] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:16.994552 10444 raft_consensus.cc:515] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:16.996240 10444 leader_election.cc:290] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819), 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:20:16.997179 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b92c2321df1b48fb93cb41d311ff2560" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923" is_pre_election: true
I20260317 21:20:16.997757 9719 raft_consensus.cc:2468] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 0.
I20260317 21:20:16.997655 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b92c2321df1b48fb93cb41d311ff2560" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4" is_pre_election: true
I20260317 21:20:16.998291 9569 raft_consensus.cc:2468] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 0.
I20260317 21:20:16.998867 9800 leader_election.cc:304] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:16.999607 10444 raft_consensus.cc:2804] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260317 21:20:16.999922 10444 raft_consensus.cc:493] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260317 21:20:17.000131 10444 raft_consensus.cc:3060] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:17.005321 10444 raft_consensus.cc:515] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:17.006855 10444 leader_election.cc:290] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 election: Requested vote from peers f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819), 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263)
I20260317 21:20:17.007781 9719 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b92c2321df1b48fb93cb41d311ff2560" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f738662691924b8a8f9b4b7cf1804923"
I20260317 21:20:17.008369 9719 raft_consensus.cc:3060] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:17.008085 9569 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b92c2321df1b48fb93cb41d311ff2560" candidate_uuid: "6c2dd7c5a524427da691785d88218218" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9b9886349405418f80c5511b013ffbd4"
I20260317 21:20:17.008777 9569 raft_consensus.cc:3060] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [term 0 FOLLOWER]: Advancing to term 1
I20260317 21:20:17.015218 9719 raft_consensus.cc:2468] T b92c2321df1b48fb93cb41d311ff2560 P f738662691924b8a8f9b4b7cf1804923 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 1.
I20260317 21:20:17.016279 9569 raft_consensus.cc:2468] T b92c2321df1b48fb93cb41d311ff2560 P 9b9886349405418f80c5511b013ffbd4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6c2dd7c5a524427da691785d88218218 in term 1.
I20260317 21:20:17.016247 9800 leader_election.cc:304] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6c2dd7c5a524427da691785d88218218, f738662691924b8a8f9b4b7cf1804923; no voters:
I20260317 21:20:17.016992 10444 raft_consensus.cc:2804] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader election won for term 1
I20260317 21:20:17.017828 10444 raft_consensus.cc:697] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [term 1 LEADER]: Becoming Leader. State: Replica: 6c2dd7c5a524427da691785d88218218, State: Running, Role: LEADER
I20260317 21:20:17.018716 10444 consensus_queue.cc:237] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } }
I20260317 21:20:17.062884 10444 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:17.063313 10444 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:17.064945 10444 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:17.066519 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:17.067361 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:17.067734 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:17.068392 10444 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:17.469378 10464 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:17.469821 10464 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:17.471297 10464 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:17.473207 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:17.473430 10465 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
W20260317 21:20:17.473695 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:17.474098 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:17.473878 10465 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:17.474720 10464 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:17.475767 10465 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:17.481686 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:17.483049 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:17.483441 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:17.484017 10465 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:17.847512 10471 consensus_queue.cc:1048] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260317 21:20:17.866070 10474 consensus_queue.cc:1048] T b92c2321df1b48fb93cb41d311ff2560 P 6c2dd7c5a524427da691785d88218218 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260317 21:20:17.992748 6633 master-stress-test.cc:577] Killing test thread 1/10
I20260317 21:20:18.033004 10482 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:18.033627 10482 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:18.035478 10482 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:18.041711 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:18.042291 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:18.042815 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:18.043761 10482 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:18.611474 10490 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:18.612016 10490 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:18.614318 10490 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:18.619827 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:18.620378 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:18.620947 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:18.621600 10490 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:18.693813 10494 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:18.694226 10494 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:18.696028 10494 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:18.698501 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:18.698987 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:18.699396 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:18.700412 10494 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:19.675602 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:19.676077 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:19.863432 10509 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:19.864127 10509 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:19.865768 10509 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:19.871457 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:19.872002 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:20:19.872368 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:19.873021 10509 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:20.745092 10523 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:20.745544 10523 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:20.747052 10523 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:20.749142 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:20.750995 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:20.751665 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:20.752525 10523 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:21.325978 10523 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:21.326453 10523 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:21.328310 10523 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:21.329854 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:21.330368 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:21.330737 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:21.331442 10523 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:22.473834 10551 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:22.474423 10551 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:22.476140 10551 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:22.478513 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 9 similar messages]
W20260317 21:20:22.481631 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:22.482069 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:20:22.482475 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:22.483127 10551 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:23.143851 10561 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:23.144464 10561 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:23.147509 10561 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:23.149500 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:23.150067 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:23.150342 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:23.150895 10561 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:23.934036 10572 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:23.934856 10572 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:23.938045 10572 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:23.939630 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:20:23.940591 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:23.940992 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:23.941756 10572 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:20:24.017506 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000e1620 after lost signal to thread 9480
W20260317 21:20:24.018424 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1b40 after lost signal to thread 9614
I20260317 21:20:24.679422 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:24.679920 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:25.688948 10595 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:25.689435 10595 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:25.691504 10595 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:25.697643 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:25.698791 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:25.699157 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:25.699738 10595 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:27.563345 10622 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:27.563796 10622 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:27.565476 10622 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:27.567375 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:27.567814 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:27.568239 9798 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:27.569006 10622 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:29.684387 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:29.685003 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:30.625669 10658 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:30.626322 10658 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:30.628036 10658 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:30.630628 10110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260317 21:20:30.633690 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:20:30.634145 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:20:30.634466 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:30.635070 10658 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:31.489760 10657 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:31.490303 10657 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:31.492316 10657 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:31.494766 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:31.495282 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:31.495664 9798 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:31.496302 10657 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:32.123566 10682 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:32.124359 10682 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:32.126351 10682 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:32.128029 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:32.128868 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:32.129504 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:32.130249 10682 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:34.069005 10707 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:34.069470 10707 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:34.071256 10707 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:34.072945 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:34.073477 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:34.073762 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:34.074265 10707 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:34.687871 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:34.688254 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:35.737938 10728 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:35.738560 10728 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:35.740392 10728 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:35.742296 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:20:35.742985 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:35.743337 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:35.744048 10728 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:35.927233 10721 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:35.927645 10721 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:35.929317 10721 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:35.930804 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:35.931250 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:35.931519 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:35.932024 10721 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:39.691890 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:39.692423 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:41.006147 10785 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:41.006641 10785 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:41.009078 10785 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:41.010578 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:41.010950 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:41.011294 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:41.011823 10785 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:41.626155 10785 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:41.626566 10785 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:41.628171 10785 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:41.629747 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:41.630165 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:41.630466 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:41.631103 10785 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:41.910723 10806 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:41.911248 10806 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:41.913435 10806 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:41.918504 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:20:41.921903 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:41.922339 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:20:41.922734 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:41.923408 10806 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:44.515012 10845 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:44.515601 10845 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:44.517949 10845 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:44.519744 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:20:44.520449 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:44.520766 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:44.521394 10845 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:44.695427 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:44.695873 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:46.256451 10871 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:46.257135 10871 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:46.258885 10871 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:46.260463 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:46.260991 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:46.261234 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:46.261751 10871 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:49.699383 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:49.699905 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:20:50.929548 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000ce1e0 after lost signal to thread 9776
W20260317 21:20:50.930409 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000e2380 after lost signal to thread 9909
I20260317 21:20:51.266667 10923 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:51.267395 10923 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:51.269141 10923 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:51.270941 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:51.271453 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:51.271867 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:51.272841 10923 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:20:52.554478 10955 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:52.555436 10955 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:52.558380 10955 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:52.559965 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:52.560563 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:52.560935 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:52.561571 10955 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:53.925300 10975 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:53.925787 10975 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:53.927325 10975 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:53.928942 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:20:53.929397 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:20:53.929697 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:53.930325 10975 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:54.703418 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:54.704025 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:20:54.709600 10975 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:54.710311 10975 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:54.711865 10975 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:54.713873 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:20:54.714298 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:20:54.714684 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:54.715325 10975 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:20:57.216899 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000c3360 after lost signal to thread 9776
W20260317 21:20:57.217818 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000e4620 after lost signal to thread 9909
I20260317 21:20:57.285854 11009 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:57.286402 11009 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:20:57.288610 11009 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:20:57.290557 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:20:57.291116 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:20:57.291455 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:20:57.292013 11009 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:20:57.878976 11010 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:20:57.879768 11010 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:20:57.882774 11010 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:20:57.885742 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:20:57.889760 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:20:57.890465 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:20:57.890923 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:20:57.891618 11010 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:20:58.050817 11014 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:20:58.051517 11014 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:20:58.053630 11014 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:20:58.055508 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:20:58.056171 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:20:58.056471 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:20:58.057188 11014 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:20:58.275111 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000bdf80 after lost signal to thread 9776
W20260317 21:20:58.276026 9906 debug-util.cc:398] Leaking SignalData structure 0x7b0800083d80 after lost signal to thread 9909
I20260317 21:20:59.707253 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:20:59.707670 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:03.676970 11023 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:03.677556 11023 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:03.679617 11023 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:03.681427 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:21:03.681967 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:21:03.682328 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:03.683027 11023 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:21:04.083061 10156 debug-util.cc:398] Leaking SignalData structure 0x7b080006f1e0 after lost signal to thread 10020
W20260317 21:21:04.083634 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba460 after lost signal to thread 10159
W20260317 21:21:04.651265 9910 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (127 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:21:04.729933 9615 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (128 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:21:04.751135 9766 heartbeater.cc:646] Failed to heartbeat to 127.6.122.126:32989 (97 consecutive failures): Network error: Failed to ping master at 127.6.122.126:32989: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:21:04.778820 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:04.779253 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:05.540321 11026 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:05.541069 11026 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:05.543236 11026 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:21:05.545281 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:21:05.545902 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:21:05.546257 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:21:05.546888 11026 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:09.782663 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:09.783159 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:13.790320 11040 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:13.790970 11040 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:13.793820 11040 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:13.795588 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:21:13.796123 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:21:13.796468 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:13.797151 11040 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:21:14.462273 11041 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:21:14.462782 11041 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:21:14.465049 11041 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:21:14.467201 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:21:14.470502 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:21:14.471046 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:21:14.471360 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:21:14.471949 11041 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:21:14.785619 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:14.786056 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:15.079003 11046 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:21:15.079614 11046 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:21:15.081568 11046 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:21:15.085911 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:21:15.086447 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:21:15.086846 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:21:15.087396 11046 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:21:15.781823 11045 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:15.782313 11045 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:21:15.783802 11045 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:15.785739 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:21:15.786258 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:21:15.786618 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:15.787247 11045 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:18.289726 11052 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:18.290227 11052 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:18.291772 11052 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:18.293699 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:21:18.294222 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:21:18.294582 9798 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:18.295186 11052 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:21:19.293288 11053 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:19.293879 11053 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:19.295727 11053 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:21:19.297526 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:21:19.298403 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:21:19.298815 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:21:19.299613 11053 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:19.789929 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:19.790529 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:22.355832 11060 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:22.356437 11060 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:22.358247 11060 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:22.359956 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:21:22.360452 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:21:22.360776 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:22.361349 11060 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:24.793501 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:24.793977 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:24.951467 11064 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:24.952095 11064 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:21:24.953892 11064 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:24.955461 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:21:24.955921 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:21:24.956225 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:24.956773 11064 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:29.798028 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:29.798514 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:30.520926 11076 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:30.521596 11076 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:30.523988 11076 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:21:30.526137 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:21:30.526760 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:21:30.527132 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:21:30.527863 11076 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:32.440056 11081 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:21:32.440557 11081 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:21:32.442720 11081 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:21:32.445457 10110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260317 21:21:32.449457 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:21:32.450125 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:21:32.450495 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:21:32.451097 11081 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:21:32.523396 11080 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:32.524307 11080 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:32.527083 11080 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:32.528888 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:21:32.529410 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:21:32.529683 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:32.530486 11080 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:21:34.801558 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:34.801959 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:35.595333 11088 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:35.595824 11088 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:35.597595 11088 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:35.599181 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:21:35.599599 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:21:35.599867 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:35.600360 11088 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:38.856266 11093 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:38.856776 11093 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:21:38.858472 11093 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:38.860281 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:21:38.860697 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:21:38.860975 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:38.861500 11093 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:21:39.419106 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba960 after lost signal to thread 10020
W20260317 21:21:39.419842 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba920 after lost signal to thread 10159
I20260317 21:21:39.841259 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:39.841723 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:44.845201 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:44.845714 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:45.558977 11103 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:45.559556 11103 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:45.561245 11103 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:45.563010 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:21:45.563557 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:21:45.563946 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:45.564550 11103 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:21:45.919690 11104 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:45.920320 11104 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:45.922567 11104 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:21:45.924599 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:21:45.925555 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:21:45.925906 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:21:45.926564 11104 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:21:48.186599 11109 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:21:48.187180 11109 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:21:48.190093 11109 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:21:48.192751 10110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:21:48.195971 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:21:48.196444 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:21:48.196820 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:21:48.197556 11109 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:21:48.762677 11113 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:48.763177 11113 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:48.765491 11113 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:48.767201 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:21:48.767870 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:21:48.768260 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:48.768875 11113 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:21:49.848907 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:49.849360 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:54.854326 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:54.855027 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:55.118716 11142 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:55.119318 11142 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:21:55.121205 11142 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:55.122762 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:21:55.123374 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:21:55.123628 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:55.124429 11142 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:21:55.876354 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000e1ba0 after lost signal to thread 9627
W20260317 21:21:55.877497 9760 debug-util.cc:398] Leaking SignalData structure 0x7b080006f620 after lost signal to thread 9763
I20260317 21:21:57.031385 11152 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:21:57.031886 11152 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:21:57.033731 11152 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:21:57.035908 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:21:57.036432 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:21:57.036880 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:21:57.037629 11152 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:21:57.780588 9906 debug-util.cc:398] Leaking SignalData structure 0x7b080003f260 after lost signal to thread 9776
W20260317 21:21:57.781749 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000b3fc0 after lost signal to thread 9909
I20260317 21:21:59.859639 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:21:59.860261 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:21:59.999392 11172 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:21:59.999881 11172 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:22:00.001745 11172 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:22:00.003477 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:22:00.007477 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:22:00.008169 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:22:00.008488 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:22:00.009300 11172 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:22:01.206650 11180 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:01.207289 11180 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:01.209350 11180 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:22:01.211261 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:22:01.211778 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:22:01.212165 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:22:01.212870 11180 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:04.863749 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:04.864230 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:22:04.994984 9910 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (247 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:22:05.027683 9615 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (248 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:22:05.038584 9766 heartbeater.cc:646] Failed to heartbeat to 127.6.122.126:32989 (217 consecutive failures): Network error: Failed to ping master at 127.6.122.126:32989: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:22:05.620448 11206 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:05.621335 11206 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:05.623567 11206 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:05.625314 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:05.626186 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:05.626497 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:05.627151 11206 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:22:06.174605 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000d4ec0 after lost signal to thread 9627
W20260317 21:22:06.175808 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000df380 after lost signal to thread 9763
W20260317 21:22:07.262323 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000d3860 after lost signal to thread 9627
W20260317 21:22:07.263233 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc1c0 after lost signal to thread 9763
I20260317 21:22:09.427007 11226 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:09.427507 11226 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:09.429103 11226 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:09.430933 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:22:09.431357 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:22:09.431670 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:09.432224 11226 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:09.754150 10113 maintenance_manager.cc:419] P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=0.033391
I20260317 21:22:09.861845 10112 maintenance_manager.cc:643] P ed19cc1cc9e74a0d9b8dc4a609f1bea9: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.107s user 0.054s sys 0.014s Metrics: {"bytes_written":13885,"cfile_init":1,"drs_written":1,"lbm_read_time_us":239,"lbm_reads_lt_1ms":4,"lbm_write_time_us":2888,"lbm_writes_lt_1ms":30,"peak_mem_usage":0,"rows_written":26,"thread_start_us":610,"threads_started":1,"wal-append.queue_time_us":960}
I20260317 21:22:09.864519 10113 maintenance_manager.cc:419] P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1360 bytes on disk
I20260317 21:22:09.866894 10112 maintenance_manager.cc:643] P ed19cc1cc9e74a0d9b8dc4a609f1bea9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":372,"lbm_reads_lt_1ms":4}
I20260317 21:22:09.868724 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:09.869185 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:22:12.440811 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000d8460 after lost signal to thread 9480
W20260317 21:22:12.441749 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1be0 after lost signal to thread 9614
I20260317 21:22:14.784044 11258 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:22:14.784662 11258 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:22:14.786399 11258 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:22:14.789158 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:22:14.792210 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:22:14.792738 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:22:14.793073 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:22:14.793663 11258 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:22:14.840353 11257 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:14.841213 11257 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:22:14.843482 11257 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:14.844929 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:22:14.845605 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:22:14.846030 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:14.846628 11257 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:14.871475 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:14.872030 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:19.026216 11284 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:19.026916 11284 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:19.029145 11284 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:22:19.031314 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:22:19.031867 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:22:19.032359 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:22:19.033090 11284 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:19.875981 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:19.876523 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:22:22.062438 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb5a0 after lost signal to thread 10020
W20260317 21:22:22.063058 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba480 after lost signal to thread 10159
I20260317 21:22:22.572180 11308 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:22.572947 11308 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:22.575349 11308 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:22.577179 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:22.577785 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:22.578145 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:22.578766 11308 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:22:23.353976 11315 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:22:23.354633 11315 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:22:23.356762 11315 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:22:23.358701 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:22:23.361788 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:22:23.362318 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:22:23.362627 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:22:23.363226 11315 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:22:23.680819 11308 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:23.681308 11308 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:23.682883 11308 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:23.684477 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:22:23.684942 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:22:23.685214 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:23.685732 11308 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:24.945768 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:24.946280 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:27.718842 11360 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:27.719537 11360 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:27.721225 11360 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:22:27.723230 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:22:27.723937 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:22:27.724244 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:22:27.724893 11360 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:29.404115 11372 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:29.404888 11372 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:29.406929 11372 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:29.408563 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:22:29.409175 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:22:29.409528 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:29.410151 11372 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:29.950368 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:29.950928 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:30.576130 11381 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:30.576759 11381 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:30.578413 11381 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:30.579965 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:30.580458 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:30.581014 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:30.581593 11381 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:22:31.374630 11394 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:22:31.375244 11394 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:22:31.379544 11394 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:22:31.382239 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:22:31.386089 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:22:31.386714 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:22:31.387014 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:22:31.387630 11394 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:22:32.820227 11413 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:32.821089 11413 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:32.823721 11413 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:22:32.825964 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:22:32.826833 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:22:32.827251 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:22:32.828362 11413 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:33.674525 11414 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:33.675211 11414 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:22:33.677210 11414 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:33.679539 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:22:33.680728 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:22:33.684506 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:33.685525 11414 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:34.954838 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:34.955329 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:36.484261 11441 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:36.484902 11441 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:36.487116 11441 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:36.489413 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:36.490314 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:36.490856 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:36.491663 11441 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:22:39.858389 11463 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:39.858856 11463 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:39.860898 11463 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:39.865010 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:39.865463 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:39.865741 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:39.866294 11463 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:22:39.959467 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:39.960141 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:40.965022 11473 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:22:40.965793 11473 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:22:40.967723 11473 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:22:40.970460 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:22:40.974386 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:22:40.975703 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:22:40.976130 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:22:40.977200 11473 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:22:44.964788 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:44.965268 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:45.152558 11511 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:45.153344 11511 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:45.155289 11511 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:45.157425 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:22:45.158001 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:22:45.158298 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:45.159057 11511 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:45.387239 11511 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:45.387861 11511 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:22:45.389823 11511 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:45.391747 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:22:45.392237 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:22:45.392578 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:45.393299 11511 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:49.969717 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:49.970290 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:52.178354 11570 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:52.179890 11570 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:52.183547 11570 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:52.186900 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:22:52.188004 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:22:52.188715 9798 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:52.189862 11570 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:52.626250 11580 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:52.627094 11580 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:52.629945 11580 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:22:52.632233 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:22:52.633040 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:22:52.633477 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:22:52.634276 11580 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:22:53.110141 11585 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:53.110647 11585 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:53.112509 11585 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:53.114174 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:53.114691 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:53.114991 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:53.115581 11585 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:22:54.973673 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:54.974152 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:22:55.475963 11607 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:22:55.476768 11607 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:22:55.479121 11607 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:22:55.483217 10110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:22:55.487914 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:22:55.488801 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:22:55.489264 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:22:55.490031 11607 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:22:57.330853 11622 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:22:57.331441 11622 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:22:57.334321 11622 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:22:57.335944 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:22:57.336440 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:22:57.336843 9798 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:22:57.337534 11622 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260317 21:22:59.001039 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000dbb00 after lost signal to thread 9627
W20260317 21:22:59.002039 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000837e0 after lost signal to thread 9763
I20260317 21:22:59.978448 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:22:59.979092 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:00.630635 11653 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:00.631251 11653 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:23:00.633219 11653 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:00.635289 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:23:00.635900 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:23:00.636351 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:00.637101 11653 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:03.940696 11687 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:03.941285 11687 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:03.943006 11687 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:03.945075 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:23:03.945724 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:23:03.946116 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:03.946743 11687 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:23:04.982728 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:04.983317 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:23:05.330212 9615 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (368 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:23:05.343163 9766 heartbeater.cc:646] Failed to heartbeat to 127.6.122.126:32989 (337 consecutive failures): Network error: Failed to ping master at 127.6.122.126:32989: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:23:05.462534 9912 heartbeater.cc:646] Failed to heartbeat to 127.6.122.126:32989 (336 consecutive failures): Network error: Failed to ping master at 127.6.122.126:32989: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:23:06.459653 11708 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:06.460247 11708 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:06.462095 11708 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:06.464170 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:23:06.464630 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:23:06.464972 9798 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:06.465613 11708 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:06.560039 11713 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:06.560797 11713 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:06.562844 11713 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:23:06.564741 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:23:06.565510 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:23:06.565974 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:23:06.566928 11713 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:08.366076 11727 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:08.366772 11727 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:08.368880 11727 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:08.370509 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:23:08.370914 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:23:08.371284 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:08.371977 11727 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:23:09.987910 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:09.988550 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:10.242110 11747 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:23:10.243003 11747 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:23:10.245424 11747 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:23:10.247259 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:23:10.250891 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:23:10.251632 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:23:10.252125 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:23:10.252852 11747 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:23:14.991760 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:14.992208 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:17.266697 11812 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:17.267405 11812 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:23:17.269889 11812 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:17.272037 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:23:17.272657 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:23:17.272953 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:17.273572 11812 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:17.326797 11812 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:17.327271 11812 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:17.329052 11812 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:17.330405 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:23:17.330863 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:23:17.331125 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:17.331619 11812 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:19.996384 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:19.997057 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:24.311434 11875 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:24.312147 11875 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:24.315053 11875 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:23:24.317018 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:23:24.317695 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:23:24.318086 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:23:24.318766 11875 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:24.536724 11878 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:24.537483 11878 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:24.539181 11878 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:24.540845 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:23:24.541443 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:23:24.541743 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:24.542366 11878 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:23:24.999970 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:25.000419 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:27.601351 11907 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:23:27.602001 11907 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:23:27.604508 11907 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:23:27.607650 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:23:27.610476 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:23:27.610909 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:23:27.611256 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:23:27.611879 11907 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:23:30.004238 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:30.004796 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:33.864180 11965 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:33.864871 11965 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:33.867111 11965 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:33.869509 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:23:33.870273 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:23:33.870635 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:33.871402 11965 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:35.008113 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:35.008544 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:37.174391 11995 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:37.174916 11995 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:23:37.176501 11995 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:37.178269 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:23:37.178788 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:23:37.179050 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:37.179590 11995 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:38.366438 12006 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:38.367460 12006 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:38.370038 12006 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:23:38.371699 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:23:38.372380 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:23:38.372757 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:23:38.373899 12006 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:40.013607 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:40.014305 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:40.483325 12019 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:40.483951 12019 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:40.485749 12019 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:40.487390 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:23:40.487949 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:23:40.488255 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:40.488862 12019 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:23:42.489712 12042 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:23:42.490490 12042 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:23:42.494557 12042 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:23:42.498672 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:23:42.506489 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:23:42.508220 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:23:42.508764 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:23:42.509594 12042 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 21:23:42.743769 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb700 after lost signal to thread 10020
W20260317 21:23:42.744334 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000bb460 after lost signal to thread 10159
I20260317 21:23:44.602164 12063 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:44.602869 12063 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:44.605419 12063 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:44.607051 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:23:44.607860 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:23:44.608171 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:44.608964 12063 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:45.053570 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:45.053962 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:46.189277 12073 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:46.189807 12073 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:46.191438 12073 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:46.193312 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:23:46.193785 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:23:46.194113 9798 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:46.194854 12073 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:23:49.388273 12107 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:49.388973 12107 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:49.391310 12107 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:23:49.393316 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:23:49.393973 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:23:49.394271 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:23:49.394865 12107 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:50.057845 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:50.058526 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:23:50.169243 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000da680 after lost signal to thread 9480
W20260317 21:23:50.170471 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000bef80 after lost signal to thread 9614
I20260317 21:23:51.118060 12121 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:23:51.118590 12121 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:23:51.120532 12121 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:23:51.123234 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:23:51.126026 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:23:51.126467 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:23:51.126816 10107 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:23:51.127527 12121 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
W20260317 21:23:54.822144 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000b9fe0 after lost signal to thread 9776
W20260317 21:23:54.823175 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000d6660 after lost signal to thread 9909
I20260317 21:23:55.062328 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:23:55.062728 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:23:55.317997 12150 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:55.318595 12150 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:23:55.320940 12150 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:23:55.323112 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:23:55.323808 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:23:55.324311 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:23:55.325136 12150 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:23:58.095227 12166 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:23:58.096035 12166 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:23:58.098192 12166 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:23:58.100457 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:23:58.101187 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:23:58.101576 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:23:58.102763 12166 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:00.067571 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:00.068116 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:00.670984 12187 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:00.671466 12187 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:00.673236 12187 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:00.674989 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:24:00.675390 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:24:00.675653 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:00.676187 12187 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:24:01.816747 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000cf700 after lost signal to thread 9480
W20260317 21:24:01.817699 9611 debug-util.cc:398] Leaking SignalData structure 0x7b08000dbe00 after lost signal to thread 9614
I20260317 21:24:03.353108 12210 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:03.353724 12210 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:03.355443 12210 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:03.357281 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:24:03.357833 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:24:03.358189 9798 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:03.358865 12210 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:24:03.693145 12213 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:03.693749 12213 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:03.695981 12213 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:03.697610 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:03.698244 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:03.698589 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:03.699334 12213 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:05.103583 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:05.104007 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:24:05.620981 9615 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (488 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:24:05.645121 9766 heartbeater.cc:646] Failed to heartbeat to 127.6.122.126:32989 (457 consecutive failures): Network error: Failed to ping master at 127.6.122.126:32989: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:05.843572 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000bba20 after lost signal to thread 10020
W20260317 21:24:05.844213 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba8e0 after lost signal to thread 10159
W20260317 21:24:05.912283 9910 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (487 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:24:06.726217 12240 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:24:06.726797 12240 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:24:06.728951 12240 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:24:06.731086 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:24:06.734870 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:06.735760 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:24:06.736095 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:24:06.737129 12240 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:24:08.187743 12257 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:08.188285 12257 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:08.190474 12257 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:08.192271 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:24:08.192726 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:24:08.193099 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:08.193779 12257 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:09.973521 12274 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:24:09.974306 12274 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:24:09.976889 12274 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:24:09.983603 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:09.984392 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:24:09.984913 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:24:09.985936 12274 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:24:10.106727 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:10.107203 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:13.684474 12308 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:13.685227 12308 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:13.687901 12308 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:13.690007 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:13.690635 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:13.691033 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:13.691658 12308 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:14.003621 12311 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:14.004496 12311 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:24:14.008672 12311 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:14.010159 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:24:14.010741 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:24:14.011297 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:14.012393 12311 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:15.110103 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:15.110471 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:18.426188 12344 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:18.426617 12344 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:18.428519 12344 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:18.430349 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:18.430779 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:18.431069 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:18.431591 12344 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:24:18.555028 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000cfea0 after lost signal to thread 9627
W20260317 21:24:18.556130 9760 debug-util.cc:398] Leaking SignalData structure 0x7b0800083900 after lost signal to thread 9763
I20260317 21:24:20.114544 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:20.115154 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:24:20.185556 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000c98c0 after lost signal to thread 9627
W20260317 21:24:20.187150 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000dc0e0 after lost signal to thread 9763
I20260317 21:24:20.392587 12358 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:20.393290 12358 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:20.395084 12358 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:20.396597 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:24:20.397125 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:24:20.397418 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:20.398018 12358 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:24:21.205243 12358 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:21.205770 12358 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:21.207963 12358 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:21.209422 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:24:21.209954 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:24:21.210326 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:21.210968 12358 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:24:21.353757 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000d1e40 after lost signal to thread 9776
W20260317 21:24:21.354854 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000d5700 after lost signal to thread 9909
I20260317 21:24:21.692487 12370 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:21.693195 12370 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:21.695144 12370 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:21.697250 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:21.698009 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:21.698303 9651 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:21.698945 12370 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:25.118542 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:25.118978 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:26.163396 12410 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:24:26.163964 12410 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:24:26.166118 12410 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:24:26.168853 10110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:24:26.168853 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 3 similar messages]
W20260317 21:24:26.172331 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:26.172773 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:24:26.173148 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:24:26.173777 12410 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:24:27.478709 12425 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:27.479502 12425 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:24:27.481395 12425 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:27.483570 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:24:27.484124 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:24:27.484387 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:27.485101 12425 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:28.614648 12434 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:28.615099 12434 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:28.616616 12434 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:28.618427 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:24:28.618932 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:24:28.619288 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:28.619897 12434 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:24:30.123457 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:30.124075 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:32.558816 12467 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:32.559633 12467 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:24:32.562563 12467 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:32.564322 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:24:32.565006 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:24:32.565397 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:32.566020 12467 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:24:32.874801 10156 debug-util.cc:398] Leaking SignalData structure 0x7b080006f300 after lost signal to thread 10020
W20260317 21:24:32.875532 10156 debug-util.cc:398] Leaking SignalData structure 0x7b08000ba6c0 after lost signal to thread 10159
I20260317 21:24:33.347689 12467 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:33.348189 12467 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:33.349946 12467 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:33.351631 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:24:33.352144 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:24:33.352443 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:33.353229 12467 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:35.156837 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:35.157230 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:35.309880 12490 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:24:35.310400 12490 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:24:35.312022 12490 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:24:35.313689 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:35.317227 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:35.317898 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:24:35.318172 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:24:35.318925 12490 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:24:39.992923 12527 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:39.993379 12527 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:39.994895 12527 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:39.997607 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:24:39.998086 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:24:39.998366 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:39.998896 12527 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:24:40.161942 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:40.162488 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:40.268359 12534 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:40.269141 12534 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:40.271797 12534 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:40.273438 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:40.274163 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:40.274467 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:40.275171 12534 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:45.165949 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:45.166709 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:24:45.540323 9760 debug-util.cc:398] Leaking SignalData structure 0x7b08000db280 after lost signal to thread 9627
W20260317 21:24:45.541326 9760 debug-util.cc:398] Leaking SignalData structure 0x7b080003dc80 after lost signal to thread 9763
I20260317 21:24:50.062060 12617 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:24:50.062754 12617 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:24:50.064635 12617 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:24:50.067625 10107 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:24:50.071518 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:24:50.072131 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:24:50.072448 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:24:50.073122 12617 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:24:50.170049 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:50.170509 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:51.464391 12632 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:51.465242 12632 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:24:51.467849 12632 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:51.469939 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:24:51.470955 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:24:51.471262 9798 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:51.472155 12632 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:52.097097 12632 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:52.097584 12632 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:52.099223 12632 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:52.100898 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:24:52.101300 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:24:52.101603 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:52.102272 12632 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:55.173336 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:24:55.173677 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:24:55.997654 12667 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:55.998299 12667 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:56.000723 12667 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:56.002336 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:56.003000 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:56.003298 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:56.003927 12667 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:59.183408 12689 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:59.183945 12689 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:59.185576 12689 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:24:59.187285 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:24:59.187659 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:24:59.187963 9651 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:24:59.188980 12689 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:24:59.240078 12688 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:24:59.240785 12688 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:24:59.242873 12688 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:24:59.244732 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:24:59.245343 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:24:59.245639 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:24:59.246394 12690 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:25:00.177441 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:25:00.178009 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:25:03.297153 12708 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:25:03.297950 12708 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:25:03.299896 12708 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:25:03.301709 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:25:03.302271 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:25:03.302618 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:25:03.303278 12708 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:25:05.004953 12716 raft_consensus.cc:493] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:25:05.005440 12716 raft_consensus.cc:515] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:25:05.007126 12716 leader_election.cc:290] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:25:05.009006 9798 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
W20260317 21:25:05.009577 9800 leader_election.cc:343] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 70e8dac62e08450da8b7ad233ebfcdb4
I20260317 21:25:05.009939 9800 leader_election.cc:304] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:25:05.010598 12716 raft_consensus.cc:2749] T 70e8dac62e08450da8b7ad233ebfcdb4 P 6c2dd7c5a524427da691785d88218218 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260317 21:25:05.181099 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:25:05.181514 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:25:05.914896 9615 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (608 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
W20260317 21:25:05.929381 9766 heartbeater.cc:646] Failed to heartbeat to 127.6.122.126:32989 (577 consecutive failures): Network error: Failed to ping master at 127.6.122.126:32989: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
I20260317 21:25:06.031247 12726 raft_consensus.cc:493] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:25:06.031775 12726 raft_consensus.cc:515] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } }
I20260317 21:25:06.033877 12726 leader_election.cc:290] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:25:06.035811 9798 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
W20260317 21:25:06.036350 9800 leader_election.cc:343] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 24470256f195441ea55e67378bd4d5bf
I20260317 21:25:06.036700 9800 leader_election.cc:304] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:25:06.037266 12726 raft_consensus.cc:2749] T 24470256f195441ea55e67378bd4d5bf P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260317 21:25:06.342244 9910 heartbeater.cc:646] Failed to heartbeat to 127.6.122.125:38261 (607 consecutive failures): Network error: Failed to ping master at 127.6.122.125:38261: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:25:06.685135 12734 raft_consensus.cc:493] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election (detected failure of leader d6b9687775954282a11f2cf829d24bac)
I20260317 21:25:06.685730 12734 raft_consensus.cc:515] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6b9687775954282a11f2cf829d24bac" member_type: VOTER last_known_addr { host: "127.6.122.126" port: 32989 } } peers { permanent_uuid: "a4719f316c614a9cadaf5edab834804d" member_type: VOTER last_known_addr { host: "127.6.122.125" port: 38261 } } peers { permanent_uuid: "ed19cc1cc9e74a0d9b8dc4a609f1bea9" member_type: VOTER last_known_addr { host: "127.6.122.124" port: 41015 } }
I20260317 21:25:06.687865 12734 leader_election.cc:290] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Requested pre-vote from peers d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989), a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261)
W20260317 21:25:06.690742 10110 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111) [suppressed 1 similar messages]
W20260317 21:25:06.693681 10107 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer d6b9687775954282a11f2cf829d24bac (127.6.122.126:32989): Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111)
W20260317 21:25:06.694559 10110 leader_election.cc:336] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: RPC error from VoteRequest() call to peer a4719f316c614a9cadaf5edab834804d (127.6.122.125:38261): Network error: Client connection negotiation failed: client connection to 127.6.122.125:38261: connect: Connection refused (error 111)
I20260317 21:25:06.694913 10110 leader_election.cc:304] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [CANDIDATE]: Term 5 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ed19cc1cc9e74a0d9b8dc4a609f1bea9; no voters: a4719f316c614a9cadaf5edab834804d, d6b9687775954282a11f2cf829d24bac
I20260317 21:25:06.695461 12734 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ed19cc1cc9e74a0d9b8dc4a609f1bea9 [term 4 FOLLOWER]: Leader pre-election lost for term 5. Reason: could not achieve majority
I20260317 21:25:10.185204 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:25:10.185735 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260317 21:25:11.518427 12778 raft_consensus.cc:493] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:25:11.518921 12778 raft_consensus.cc:515] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:25:11.521629 12778 leader_election.cc:290] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819)
W20260317 21:25:11.523456 9798 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
W20260317 21:25:11.523959 9800 leader_election.cc:343] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f738662691924b8a8f9b4b7cf1804923 (127.6.122.66:42819): Not found: Tablet not found: 889aad7114b04c30b394b3c9936b2abc
I20260317 21:25:11.524266 9800 leader_election.cc:304] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c2dd7c5a524427da691785d88218218; no voters: 9b9886349405418f80c5511b013ffbd4, f738662691924b8a8f9b4b7cf1804923
I20260317 21:25:11.524924 12778 raft_consensus.cc:2749] T 889aad7114b04c30b394b3c9936b2abc P 6c2dd7c5a524427da691785d88218218 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:25:12.952497 12793 raft_consensus.cc:493] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260317 21:25:12.953218 12793 raft_consensus.cc:515] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9b9886349405418f80c5511b013ffbd4" member_type: VOTER last_known_addr { host: "127.6.122.65" port: 40263 } } peers { permanent_uuid: "f738662691924b8a8f9b4b7cf1804923" member_type: VOTER last_known_addr { host: "127.6.122.66" port: 42819 } } peers { permanent_uuid: "6c2dd7c5a524427da691785d88218218" member_type: VOTER last_known_addr { host: "127.6.122.67" port: 38595 } }
I20260317 21:25:12.955641 12793 leader_election.cc:290] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263), 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595)
W20260317 21:25:12.957453 9651 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 9b9886349405418f80c5511b013ffbd4 (127.6.122.65:40263): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
W20260317 21:25:12.958225 9652 leader_election.cc:343] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 6c2dd7c5a524427da691785d88218218 (127.6.122.67:38595): Not found: Tablet not found: 841735a56c4c4df7a58f93ddbaf67a09
I20260317 21:25:12.958498 9652 leader_election.cc:304] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f738662691924b8a8f9b4b7cf1804923; no voters: 6c2dd7c5a524427da691785d88218218, 9b9886349405418f80c5511b013ffbd4
I20260317 21:25:12.959079 12793 raft_consensus.cc:2749] T 841735a56c4c4df7a58f93ddbaf67a09 P f738662691924b8a8f9b4b7cf1804923 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260317 21:25:15.188870 9972 master_runner.cc:186] Error getting master registration for 127.6.122.126:32989: Network error: Client connection negotiation failed: client connection to 127.6.122.126:32989: connect: Connection refused (error 111),
I20260317 21:25:15.189253 9972 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260317 21:25:16.400902 9906 debug-util.cc:398] Leaking SignalData structure 0x7b0800083d20 after lost signal to thread 9776
W20260317 21:25:16.401893 9906 debug-util.cc:398] Leaking SignalData structure 0x7b08000db780 after lost signal to thread 9909
I20260317 21:25:16.664987 10113 maintenance_manager.cc:419] P ed19cc1cc9e74a0d9b8dc4a609f1bea9: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1360 bytes on disk
I20260317 21:25:16.678871 10112 maintenance_manager.cc:643] P ed19cc1cc9e74a0d9b8dc4a609f1bea9: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.011s sys 0.001s Metrics: {"dirs.queue_time_us":1110,"thread_start_us":1277,"threads_started":2,"wal-append.queue_time_us":967}
W20260317 21:25:17.134306 9922 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
W20260317 21:25:17.134867 9922 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
W20260317 21:25:17.135258 9922 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
W20260317 21:25:17.135492 9922 master_proxy_rpc.cc:177] Failed to determine new leader Master: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
F20260317 21:25:17.146966 9931 master-stress-test.cc:297] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_9da3262f4c254404acf6cd7ddac1377c on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
F20260317 21:25:17.148676 9933 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
*** Check failure stack trace: ***
*** Aborted at 1773782717 (unix time) try "date -d @1773782717" if you are using GNU date ***
*** Check failure stack trace: ***
PC: @ 0x0 (unknown)
*** SIGABRT (@0x3e8000019e9) received by PID 6633 (TID 0x7f20ba2fb700) from PID 6633; stack trace: ***
@ 0x2de460 __tsan::CallUserSignalHandler() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:1915
@ 0x2e0943 rtl_sigaction() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:2005
@ 0x7f20cd403980 (unknown) at ??:0
@ 0x7f20c85b0e87 gsignal at ??:0
@ 0x7f20c85b27f1 abort at ??:0
@ 0x2e4456 abort at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:263
@ 0x7f20cb648da4 google::LogMessage::SendToLog() at ??:0
@ 0x7f20cb649950 google::LogMessage::Flush() at ??:0
@ 0x7f20cb64ea8b google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x360594 kudu::MasterStressTest::CreateTableThread() at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/master-stress-test.cc:?
@ 0x38e45e _ZNSt3__18__invokeIMN4kudu16MasterStressTestEFvvEPNS1_26MasterStressTest_Test_TestEJEvEEDTcldsdeclsr3std3__1E7forwardIT0_Efp0_Efp_spclsr3std3__1E7forwardIT1_Efp1_EEEOT_OS7_DpOS8_ at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/type_traits:3846
@ 0x38e311 _ZNSt3__116__thread_executeINS_10unique_ptrINS_15__thread_structENS_14default_deleteIS2_EEEEMN4kudu16MasterStressTestEFvvEJPNS6_26MasterStressTest_Test_TestEEJLm2EEEEvRNS_5tupleIJT_T0_DpT1_EEENS_15__tuple_indicesIJXspT2_EEEE at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:281
@ 0x38da5b std::__1::__thread_proxy<>() at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/installed/tsan/include/c++/v1/thread:?
@ 0x2de320 __tsan_thread_start_func at /home/jenkins-slave/workspace/build_and_test_flaky@2/thirdparty/src/llvm-11.0.0.src/projects/compiler-rt/lib/tsan/rtl/tsan_interceptors_posix.cpp:959
@ 0x7f20cd3f86db start_thread at ??:0
F20260317 21:25:17.148676 9933 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: AlterTable timed out after deadline expired: AlterTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)F000F20260317 21:25:17.177387 9934 master-stress-test.cc:328] 000F20260317 21:25:17.157599 9930 master-stress-test.cc:328] Check failed: _s.ok() Bad status: Timed out: Error creating table default.Table_c25a4ee82851411b8227cf1f4f04844e on the master: CreateTable timed out after deadline expired: CreateTable passed its deadline: Timed out: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 768) passed its deadline: Not found: no leader found: ConnectToClusterRpc(addrs: 127.6.122.126:32989,127.6.122.125:38261,127.6.122.124:41015, num_attempts: 1)
*** Check failure stack trace: ***
@ 0x7f20c869361f clone at ??:0